Degraded performance across Vero platform

Incident Report for Vero

Postmortem

Overnight, shortly before 8:00AM UTC, our internal monitoring system alerted us to issues with our API processing. Upon investigation we realized that API processing, email processing, and several other services (including access to the UI) were being impacted.

Investigation revealed that an unusual pattern of heavy API traffic was the cause. Actions were taken to mitigate the issue, and we were able to restore functionality to our services, and begin processing the backlog. At approximately 1:57PM UTC, the backlog had been fully processed and all API requests and emails began to be processed in real time.

We'll be looking into how to optimize and guard against this particular pattern of API traffic in the future.

Posted Sep 02, 2022 - 09:36 AEST

Resolved

This incident has been resolved. All API requests and emails are being processed in real time. If you experience any further issues please email support@getvero.com and we will investigate ASAP.

Posted Sep 01, 2022 - 23:57 AEST

Monitoring

API requests are now processing 🎉. All services are now operational and we are working to process a backlog of requests. Once all data is processing real time we will resolve the issue and follow up with additional details.

As always, please don't hesitate to email support@getvero.com if you have any questions

Posted Sep 01, 2022 - 20:57 AEST

Update

Emails are now processing. We expect there will still be some delays in sending emails whilst we process the current backlog. If you have any questions regarding your campaigns please don't hesitate to email support@getvero.com.

Posted Sep 01, 2022 - 19:45 AEST

Identified

The UI is now accessible! We've also identified the root cause and should be able to process API requests and emails shortly.

Posted Sep 01, 2022 - 19:30 AEST

Investigating

We're currently seeing delays in processing API requests and emails as well as loading the UI.

Our team has been working to mitigate these issues and we will provide further updates as soon as possible. Please don't hesitate to contact support@getvero.com if you have any questions.

Posted Sep 01, 2022 - 19:10 AEST

This incident affected: Vero Cloud: Ingestion API, Vero Cloud: Newsletter processing, Vero Connect: Newsletter processing, Vero Connect: UI, Vero Cloud: UI (General UI access and speed), and Vero Cloud: Automated email processing (Transactional emails, Behavioral emails, Workflows).