Delays in Newsletter sending
Incident Report for Vero
Postmortem

At 10:45 UTC 12 June we received alerts that some customers' newsletters were delayed.

Our team responded, made adjustments and reviewed our systems. Reporting confirmed that newsletters were sending as normal.

At 16:00 UTC we were again alerted to delays in newsletter processing. Our team re-investigated and found that one of our database clusters was no longer performant. This issue affected our ability to calculate segments and schedule newsletters, leading to delays for the majority of newsletters scheduled between 10:45 UTC and 17:30 UTC.

Our database was affected as a result of regular maintenance processes we run. Unfortunately this issue both affected scheduling and the regular health checks that report on newsletter delays.

Since 18:00 UTC, newsletters have been sending as normal.

Data collectiong, workflows and automated messages were not affected by this incident.

Based on what we have observed from this incident we will make changes to our maintenance processes and health checks.

If you have any questions, please conact us at support@getvero.com. Thank you.

Posted Jun 13, 2021 - 13:10 AEST

Resolved
Newsletters have been sending as normal since approximately 18:00 UTC 12 June.
Posted Jun 13, 2021 - 13:10 AEST
Monitoring
We are now processing newsletters again as normal but have a large backlog to work through. This will take us some time. We will continue to work through this backlog and monitor this issue until we're comfortable everything has caught up.

We will update this incident to "Resolved" at that time (or post further updates if there are any other changes in state).
Posted Jun 13, 2021 - 03:29 AEST
Update
We believe we have identified the cause of the newsletter sending delays. We are currently working through a backlog of delayed sends.
Posted Jun 13, 2021 - 03:05 AEST
Identified
We have observed delays in sending newsletters and are investigating the root cause of this issue.
Posted Jun 12, 2021 - 20:00 AEST
This incident affected: Newsletter processing.