On Monday 16th of May at 9am UTC we experienced performance degradation in our event stream processing service.
As we process events at a significant volume this most notably led to delays in processing emails and statistics whilst we restored performance.
Once the issue was solved it took some time to process the backlog which completed on Tuesday 17th of May at 8:12am UTC.
Posted May 20, 2022 - 15:52 AEST
Update
All has been processing as expected for several hours and we're now confident the issue is resolved. We'll follow up shortly with a full post-mortem.
Posted May 18, 2022 - 17:02 AEST
Monitoring
All services are now operational and we are working to process a backlog of requests.
It's possible we will still experience some intermittent delays over the next 24 hours, as we continue to solve the root cause. Until then we'll keep components marked as degraded.
Our support team will be reaching out to customers whose newsletters have been affected.
Please email support@getvero.com if you have any questions regarding your account.
Posted May 17, 2022 - 16:12 AEST
Investigating
We are currently seeing a degradation in performance which has predominantly affected automated email processing. Further impacted systems that have seen intermittent delays are calculating segments, processing newsletters as well as the campaign statistics, logs and webhooks that result from sending emails.
Our team has been working to mitigate these issues and we will provide further updates shortly - we will ofcourse write a full post-mortem once all is resolved. Please don't hesitate to contact support@getvero.com if you have any questions.
Posted May 17, 2022 - 09:03 AEST
This incident affected: Vero 1.0: Reports data availability (Reports data availability (Vero default and Mailgun integrations), Reports data availability (Sendgrid and other non-Mailgun integrations)), Vero 1.0: Automated email processing (Transactional emails, Workflows, Behavioral emails), and Vero 1.0: Segment calculation, Vero 1.0: Newsletter processing, Vero 2.0: Newsletter processing, Vero 2.0: Reports data availability.