Email Log Delays and Decreased Newsletter Throughput
Incident Report for Vero
Resolved
We are closing this incident as this issue should now be resolved.

We are likely to provide a post mortem in the near future.
Posted May 04, 2017 - 13:33 AEST
Update
Hi all,

We hope your week is off to a good start.

Over the weekend we were successfully able to return out systems to normal processing. Since Sunday we have been monitoring our systems to ensure things are operating as we would expect.

We have seen several instances of degraded performance, experiencing the same degredation as in our last update.

We're fine-tuning a few things and hope to have this issue closed out in the next 24-48 hours.

If you are experiencing issues or have questions about any specific impact on your account, feel free to email us at support@getvero.com.

Thanks!
Posted May 03, 2017 - 00:23 AEST
Update
We have made solid progress in restoring our systems to full capacity, predominantly through our own work and deployment of failover services. In the near future we will no longer rely solely on the affected provider, thanks to some broader infrastructure projects we have in place.

The system has been operating essentially as normal over the last 36 hours. As an update, there are currently two areas being affected:

1. Conversion tracking. Conversion tracking is delayed by an estimate of 48 hours at this time. This results in both conversion webhooks and dashboard statistics being underreported. Our aim is to have conversions caught up and in realtime by Monday evening.
2. Behavioral email delays. We are seeing some periods where behavioral emails are delayed. The extent of this fluctuates when it arises. When this is observed, we are doing all we can to assign extra resources and ensure emails are sent without any delay. Our Operations Team is watching this explicitly.

By Monday we anticipate that our service provider will have completely resolved the issues we have been experiencing and that we will be able to proceed at our usual operating capacity.

Apologies for the inconvenience caused. Thanks again for your patience in scenarios where these issues have affected you.

Have a great Friday wherever you are! 🌏
Posted Apr 28, 2017 - 15:45 AEST
Update
We wanted to provide an update.

Today we have seen reasonable performance across our application except for our Email Logs. Unfortunately, at this time, there are two components still affected:

• Email Logs are delayed in the UI by 12+ hours. This is the largest impact.
• Conversion figures are being processed but are delayed.

We continue to work with our provider through the issues here. We have made progress and hope to have a better idea of an ETA within the next few hours. We will provide an update at that time.

We apologise for the inconvenience caused. If you would like us to review or check any details regarding email sends, please email us at support@getvero.com – we would be more than happy to help. Thank you.
Posted Apr 25, 2017 - 16:43 AEST
Monitoring
At 0400 UTC this morning, a component of our application experienced an outage. At this time, several users reported experiencing trouble connecting to our web application (app.getvero.com).

We were able to resolve the issues with the infrastructure component in under 30 minutes and our web application returned to a functioning state.

Over the last several hours we have begun to observe degraded performance from this component. As this component is important to the processing of several elements at Vero we are currently seeing this affect two main areas:

1. **Email Logs (app.getvero.com/logs)** are delayed. This is affecting customers' ability to see updates related to email status in realtime in our UI.
2. **Newsletter throughput** is less than normal. We will be monitoring to ensure the send rate does not drop beyond workable levels.

We're currently in discussions with the provider of the infrastructure component that is experiencing degraded performance and we are investigating the root cause. In the meantime we're re-routing processing with an aim to get things back to normal speed.

Note that our API collection and processing remains unaffected and the other components of our UI should be usable as normal.

We're doing everything we can to ensure your experience of Vero is not noticeably affected and we hope to have things operating 100% in the near future.

We'll provide relevant updates here. If you have specific questions, please email us at support@getvero.com, our team will be happy to answer them.
Posted Apr 25, 2017 - 00:48 AEST