Delays in all queue processing
Incident Report for Vero
Resolved
This incident has been resolved.
Posted 2 months ago. Aug 09, 2017 - 12:41 AEST
Update
We intend to mark this issue as resolved shortly.

A brief timeline of this issue:

• 23:15 (8 August) - 23:50 UTC: we began observing instability with one of our core pieces of infrastructure
• 23:50 (8 August) - 00:20 (9 August) UTC: a partial outage and delays in process all queues.
• 00:20 (9 August) UTC: processing fully restored

Since this time we have been processing normally, however we have been working through a backlog of behavioral, transactional and newsletter emails delayed since 23:50. This is ongoing. We do not yet have the full impact of these delays, but some emails will have been delayed for around one hour.

We will provide a further update once this is fully caught up. At that time we'll mark this as resolved.
Posted 2 months ago. Aug 09, 2017 - 11:05 AEST
Monitoring
The system has been operational now for about 15 minutes. We're continuing to monitor to ensure all is running as normal.

We have a small backlog to process, but this should be caught up shortly.
Posted 2 months ago. Aug 09, 2017 - 09:39 AEST
Identified
We have observed delays across all of our processing queues. We have identified that this relates to one of our core pieces of queuing infrastructure.

We are working on restoring full processing capacity and anticipate this won't take too long. We will report further updates here as we have them and have updated the Component statuses above to reflect the current situation.
Posted 2 months ago. Aug 09, 2017 - 08:50 AEST