• 23:15 (8 August) - 23:50 UTC: we began observing instability with one of our core pieces of infrastructure
• 23:50 (8 August) - 00:20 (9 August) UTC: a partial outage and delays in process all queues.
• 00:20 (9 August) UTC: processing fully restored
Since this time we have been processing normally, however we have been working through a backlog of behavioral, transactional and newsletter emails delayed since 23:50. This is ongoing. We do not yet have the full impact of these delays, but some emails will have been delayed for around one hour.
We will provide a further update once this is fully caught up. At that time we'll mark this as resolved.
Posted Aug 09, 2017 - 11:05 AEST
Monitoring
The system has been operational now for about 15 minutes. We're continuing to monitor to ensure all is running as normal.
We have a small backlog to process, but this should be caught up shortly.
Posted Aug 09, 2017 - 09:39 AEST
Identified
We have observed delays across all of our processing queues. We have identified that this relates to one of our core pieces of queuing infrastructure.
We are working on restoring full processing capacity and anticipate this won't take too long. We will report further updates here as we have them and have updated the Component statuses above to reflect the current situation.