Between 18:30 and 19:20 UTC today, Vero's web application was unavailable.
Whilst our API remained fully operational during this period, we also experienced delays processing Newsletters and all automated campaigns.
One of our service providers was conducting known, scheduled maintenance to one of our production data stores, responsible for managing company authentication, amongst other things. This maintenance followed a pattern of maintenance that has occurred before and we anticipated no key risks with the maintenance.
After the maintenance was complete, our application could not get a connection to our primary, or replica, data stores hosted at this provider. Our Operations Team intervened immediately but was not able to find a stable solution until 19:20.
At this time, Vero's web application began operating as normal and we processed the backlog of campaigns that had built up. Full realtime processing was restored by 19:40 UTC.
As a result of this issue, we will be conducting a review of our alerting and rollover procedures, and consider ways in which we might leverage alternative service providers to increase redundancy for this particular infrastructure component.
We want to confirm that no data was lost and, ultimately, all emails were processed.
If you have any questions, please let us know via email at firstname.lastname@example.org. Many thanks.