API processing delays
Incident Report for Vero
Postmortem

Earlier today we experienced errors with our API processing.

We apologise for the inconvenience caused. We understand the critical importance of ensuring all API requests are processed without error. As a result, we wanted to share a more detailed post-mortem regarding this incident.

The following key points provide the timeline and details:

• From approximately 13:30 UTC 1 May we began to see irregular spikes in the number of errors being returned by our tracking API.
• By approximately 15:20 UTC, these errors had escalated to the point that our API was returning 502 and 504 errors 50% of the time.
• Error rates persisted at around 50% between 15:20 and 16:15 UTC.
• During this time, API requests that received an error response were not processed by Vero unless they were explicitly retried. Segment.com and a portion of our API libraries automatically handle retries.
• We have determined this issue was the result of a configuration issue, resulting in select servers in our API cluster running out of memory.
• We are in the process of fixing this regression and should have the required changes live within the next several days.
• In the interim we have updated our alerting procedures to ensure there are no further repeats of this issue.

If you have any questions, please email us at support@getvero.com. As always, thank you for your business.

Posted 21 days ago. May 02, 2019 - 14:19 AEST

Resolved
This incident has been identified and resolved. The backlog of API requests was able to process shortly after our previous update and all requests are now processing in real-time.

Please email support@getvero.com if you have any questions regarding your account.
Posted 22 days ago. May 02, 2019 - 02:17 AEST
Investigating
Our API is currently experiencing delays in processing.

We are investigating the root cause and will post an update as soon as we have more information.
Posted 22 days ago. May 02, 2019 - 01:33 AEST
This incident affected: API.