Intermittent delays in processing API requests, emails, statistics and webhooks

Incident Report for Vero

Postmortem

Last week, beginning ~10:00 25 October UTC and ending around 18:00 26 October UTC (a period of around 48 hours), we experienced several periods of significant API and email processing delays.

This instability was caused by an inefficient query pattern behind one of the conditions used in workflows. This inefficiency was revealed based on a usage pattern we had not seen before.

Once identified as the root cause, we took steps to mitigate this issue and introduce guardrails to prevent it arising in the future. We monitored for several days to ensure that all was fully resolved.

We have not seen the issue since and consider the issue resolved.

We are sorry for any inconvenience caused. If you have any follow up questions please contact us at support@getvero.com.

Posted Nov 05, 2022 - 04:58 AEDT

Resolved

This incident has been resolved. We will follow up with additional details soon but confirming all API requests and emails have been processing in real-time for the past 24 hours.

In the meantime, please don't hesitate to email support@getvero.com if you have any urgent questions about your account.

Posted Oct 28, 2022 - 16:06 AEDT

Update

Over the last 48 hours (since ~10:00 25 October UTC) we've experienced several periods of significant email processing delays.

Yesterday and today (26 October UTC) our operations and engineering team have taken multiple steps to mitigate these issues. As a result of these measures we have seen improvements to processing transactional emails (as evident in the data below).

We are still investigating the root cause of these issues and will provide a post mortem once we are confident all problems have been resolved. Until this time it is possible that further delays may occur. We will continue to investigate.

To help users troubleshoot we are sharing average delay times during the affected periods. All times below are in UTC.

For transactional automated messages, **average** delays were as follows:

Tuesday 25 October
• 10:00 - 16:00 = 90 mins

Wed 26 October
• 09:00 - 18:00 = 22 minutes

For behavioural automated messages and workflows, **average** delays were as follows:

Tuesday 25 October
• 10:00 - 16:00 = 160 minutes
• 19:00 - 11:59 = 110 minutes

Wed 26 October
• 00:00 - 2:00 = 110 minutes
• 9:00 - 18:00 = 4 hours

If you have any questions please contact us at support@getvero.com.

Posted Oct 27, 2022 - 17:31 AEDT

Investigating

Over the last 30 hours we've been experiencing intermittent delays affecting API processing and resulting in transactional, behavioural, workflow and newsletter delays. We've been updating each of the components that have been affected as they occur.

As of right now API processing and emails including behaviourals, workflows and newsletters are real time. Unfortunately transactional emails are currently seeing processing delays and we're working through a backlog of statistics, logs and webhooks.

Throughout this time we've been focused on identifying the root cause and are working hard to ensure all processing remains real time. We'll be posting updates as we receive them and will follow up with a detailed post-mortem once the issue is resolved.

If you have any questions about your account please email support@getvero.com

Posted Oct 27, 2022 - 07:43 AEDT

This incident affected: Vero Cloud: Reports data availability (Reports data availability (Vero default and Mailgun integrations), Reports data availability (Sendgrid and other non-Mailgun integrations)) and Vero Cloud: Automated email processing (Transactional emails).