Logs and Customer Profiles pages not responding

Incident Report for Vero

Resolved

At 12:30 UTC yesterday, several of the nodes in the search cluster powering the Logs and Customer Profile pages in Vero went offline.

We were alerted to this issue and, by 13:20 UTC, had begun to restore the affected nodes.

Due to recovery time the UI was returned to normal at 16:30 UTC. During the intervening period the UI was unavailable. Once restored, there were delays in processing logs into the UI as the accumulated backlog caught up. No other processes were affected.

We have investigated and understood the root cause of this issue. Fortunately, we have been working toward replacing this component of our architecture over the last few months as part of planned infrastructure changes. We will be finalising this work within the new two weeks and are confident that this issue should not arise again.

If you have any questions please email us at support@getvero.com.

Posted Nov 26, 2019 - 23:30 AEDT