At 12:31 UTC 17 April we were alerted to intermittent issues with Vero's UI.
By 01:57 UTC we had identified the root cause: a recent deploy to the search cluster powering elements of our UI had led to unforeseen load on that cluster, causing it to fail. We rolled the changes back and, unfortunately, had to initiate a full recovery on the cluster involved.
UI access was restored at approximately 05:30 UTC 17 April, with significant delays in the Logs and Customers functionality whilst the restore continued.
The recovery process completed at approximately 09:45 UTC 17 April.
During the period 12:31-09:45 UTC, only UI access was interrupted. Backend processing continued as normal, with no effect on newsletters, workflows or other functions.
We are continuing to monitor to ensure the new cluster is performing as expected and there are no further interruptions.
If you have any further questions, please email us at firstname.lastname@example.org.