Conversion results have been updating in realtime for the last 48 hours and we can confirm this incident is now resolved! 🎉
Posted Mar 29, 2021 - 16:05 AEDT
Update
Messages, logs, reports and webhooks continue to be processed in realtime. We are still processing the backlog of conversion statistics which we anticipate will be complete within 6-12 hours.
If you have any questions about your account please email us at support@getvero.com.
Posted Mar 26, 2021 - 15:57 AEDT
Monitoring
All messages, logs, reports and webhooks are processing realtime. Conversion statistics will be updating in realtime within the next 12 hours.
We'll continue to monitor and provide updates as they occur.
Posted Mar 25, 2021 - 17:12 AEDT
Identified
We are still processing at a reduced rate. Reports and Logs are particularly affected, as are webhooks. We've updated the components below to reflect this.
Message throughput has been affected periodically since 1:30pm UTC (Wednesday) and we have been updating the affected components to reflect this as it occurs.
We are making progress on this issue and apologise for the extended impact.
Posted Mar 25, 2021 - 10:05 AEDT
Investigating
On Tuesday 23rd March at approximately 6:30pm UTC we observed delays in processing automated messages (behavioural, transactional and workflows).
We identified an issue with our round-robin system. We debugged and fixed the root cause by approximately 9.30pm UTC. During the period 6:30pm-9.30PM UTC automated messages were operating at a severely reduced throughput, resulting in a backlog of automated messages. The backlog of automated messages was fully processed and running in realtime by 11pm UTC. Newsletters sent between 9.30pm-12:30am UTC (Wednesday) were sent at a reduced rate whilst this backlog was processed, resulting in some delays.
We've been posting updates to the various system components on this status page over the last 12 hours to reflect the above delays as they occurred.
--
Since 12:30am UTC we have identified a second issue with some of the raw hardware provided by our cloud provider. **It appears this is a hardware issue and is not related to the earlier incident**. Messages **are** currently sending in realtime but we are operating at an overall reduced capacity with some delays in aggregating conversion metrics and sending webhooks.
We're working with our cloud provider to resolve this issue, at which time we'll be able to scale to full capacity. In the interim we're monitoring and will post updates if the situation changes and message send speeds are affected.
This incident affected: Vero Workflows: Reports data availability (Reports data availability (Vero default and Mailgun integrations), Reports data availability (Sendgrid and other non-Mailgun integrations)), Vero Workflows: Newsletter processing, and Vero Workflows: UI (Logs page activity).