Vero UI unavailable
Incident Report for Vero
Postmortem

At 12:31 UTC 17 April we were alerted to intermittent issues with Vero's UI.

By 01:57 UTC we had identified the root cause: a recent deploy to the search cluster powering elements of our UI had led to unforeseen load on that cluster, causing it to fail. We rolled the changes back and, unfortunately, had to initiate a full recovery on the cluster involved.

UI access was restored at approximately 05:30 UTC 17 April, with significant delays in the Logs and Customers functionality whilst the restore continued.

The recovery process completed at approximately 09:45 UTC 17 April.

During the period 12:31-09:45 UTC, only UI access was interrupted. Backend processing continued as normal, with no effect on newsletters, workflows or other functions.

We are continuing to monitor to ensure the new cluster is performing as expected and there are no further interruptions.

If you have any further questions, please email us at support@getvero.com.

Posted Apr 19, 2021 - 11:56 AEST

Resolved
This incident has now been resolved. We are continuing to monitor for any further issues.
Posted Apr 17, 2021 - 23:49 AEST
Update
We are continuing to monitor for any further issues.
Posted Apr 17, 2021 - 23:24 AEST
Monitoring
All parts of the Vero UI have been reliably available again over the past hour and we will continue to monitor. There will be some delays to accessing recent campaign statistics and logs data in certain sections of the Vero UI but we expect any backlog to be processed over the next few hours and will update our status components accordingly.
Posted Apr 17, 2021 - 21:53 AEST
Update
We are continuing to work on a fix for this issue.
Posted Apr 17, 2021 - 18:55 AEST
Update
We are continuing to work on a fix for this issue.
Posted Apr 17, 2021 - 16:09 AEST
Update
We are continuing to work on a fix for this issue
Posted Apr 17, 2021 - 13:07 AEST
Identified
We are have been experiencing intermittent issues causing parts of Vero's UI to be slow or unresponsive, we are currently working to resolve the issue.
Posted Apr 17, 2021 - 12:03 AEST
This incident affected: Vero Workflows: Reports data availability (Reports data availability (Vero default and Mailgun integrations), Reports data availability (Sendgrid and other non-Mailgun integrations)) and Vero Workflows: UI (General UI access and speed, Logs page activity).