After monitoring for 24 hours, we are confident this issue has been resolved. Segment calculations have returned to normal processing, and newsletters are going out without further issue.
Posted Aug 10, 2023 - 08:26 AEST
Update
We have been monitoring and the system has been stable for the last six+ hours. We'll continue to monitor for a full 12 hours before marking this incident as resolved.
Thanks again for your patience and support.
Posted Aug 08, 2023 - 19:23 AEST
Monitoring
As of 2:50am Tuesday UTC all newsletters and segments have now caught up and are processing in real time.
We are monitoring the changes we've made to ensure there are no regressions. We will continue to monitor over the next several hours and will post further updates as required.
Once we are comfortable everything is stable we will mark this incident as resolved.
Posted Aug 08, 2023 - 13:00 AEST
Identified
**At the present time all newsletters and UI-triggered segment calculations are processing at a significantly reduced rate.** Many newsletters are delayed by several hours.
We've found the root cause.
We acknowledge the impact this has and we are working as hard as we can to resolve this issue.
Further background:
• At 10pm Saturday UTC we were alerted to processing delays in segments and newsletters for a small number of customers. These were reviewed and worked through individually. • At 7am Sunday UTC we were alerted to delays for a larger portion of customers. In light of this development, our platform team began a deeper review. Affected segments and newsletters were worked through by ~Sunday 2pm UTC and we were not of the view there was a systemic issue. Changes we made appeared to have resolved the issue. • At 4pm Monday UTC we began noticing a new increase in delays. • By 7pm Monday UTC we had uncovered the root cause of this issue. • By 10pm Monday UTC we had taken action to reduce processing speeds and posted an official incident.
As of 2am Tuesday UTC (the present time) we are still processing newsletters at a reduced rate. We will post an update once we have a clear estimate of when newsletter sending will be back to normal rates.
We've identified an issue resulting in poor performance for segment calculations. It was occurring sporadically over the last 48 hours, however we now understand it to be systemic. Our platform team is working to identify the root cause, and currently investigating several possibilities. In the interim, manual segment refreshes (via the UI) have been disabled to prioritize queries from newsletters. Workflows and automations are unaffected.
Posted Aug 08, 2023 - 08:02 AEST
This incident affected: Vero Cloud: Newsletter processing and Vero Cloud: Segment calculation.