This incident has been resolved, all services are real time and operating as normal. If you have questions or want to discuss potential or observed impact on your account please email us at support@getvero.com.
Posted Oct 20, 2021 - 22:32 AEDT
Update
At 02:00 UTC today we identified processing delays with some of our workers.
After several hours of investigation and intermittent reliability issues we made the decision to pause our workers temporarily in order to determine the root cause. As a result we were able to pinpoint the cause: a regression in a code deploy made earlier today causing worker configuration issues and resulting in workers consistently failing before completing their jobs.
Once fixed we were able to redeploy our worker fleet with services resuming at 10:12 UTC. Since this time we have been working through the backlog of accumulated API calls and automations for the Vero Workflows product. Please see a summary of the affected service timeline below.
No data was lost during this incident.
**Vero Newsletters** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters. 07:50 UTC - 10:00 UTC: not able to send newsletters.
**Vero Workflows** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters (within Vero Workflows). 07:50 UTC - 10:00 UTC: - Not able to send newsletters. - No automated processing. - No API call processing (though all data was safely collected). 10:00+ UTC: working through API and automation backlog (API real time as of 9.15pm).
--
We will continue to monitor and will resolve this incident once all services are real time and operating as normal.
If you have any questions, please email us at support@getvero.com. Thank you for your patience!
Posted Oct 20, 2021 - 21:45 AEDT
Update
We are continuing to monitor for further issues but we anticipate that our system backlog should be processed in approximately another hour, we will update our status components back to Operational as those workloads are real-time again.
Posted Oct 20, 2021 - 21:09 AEDT
Monitoring
A fix has been implemented and we are currently monitoring. We expect there will be delays in processing emails whilst we process a backlog of requests over the next few hours.
Posted Oct 20, 2021 - 19:42 AEDT
Identified
The issue has been identified and a fix is being implemented
Posted Oct 20, 2021 - 18:27 AEDT
Update
We are currently experiencing delays in processing emails which our team is actively working to resolve. This includes both newsletters and now automated campaigns. If you have questions or want to discuss potential or observed impact on your account please email us at support@getvero.com.
Posted Oct 20, 2021 - 17:55 AEDT
Update
We are continuing to investigate this issue - newsletters are sending but noting we are seeing queuing delays of 30+ minutes
Posted Oct 20, 2021 - 16:45 AEDT
Investigating
We are currently investigating delays in queuing of newsletter campaigns
Posted Oct 20, 2021 - 15:56 AEDT
This incident affected: Vero Cloud: Automated email processing (Transactional emails, Behavioral emails, Workflows) and Vero Cloud: Newsletter processing, Vero Connect: Newsletter processing.