Monitoring - The issue has been resolved for the past 18 hours. We are currently conducting a review of all related emails sent during this period to ensure statistics are now up to date.
Oct 18, 19:50 AEDT
Investigating - We have identified an issue receiving webhooks from Mandrill and Sendgrid leading to a delay in updating related campaign statistics. Emails are sending on time (as expected) and once resolved all delayed statistics will be updated as quickly as possible.
If you have any questions about your campaigns please don't hesitate to contact support@getvero.com
Oct 18, 05:27 AEDT
API   ? Operational
90 days ago
99.99 % uptime
Today
Email Processing Operational
90 days ago
99.99 % uptime
Today
Transactional emails   Operational
90 days ago
99.99 % uptime
Today
Behavioral emails   Operational
90 days ago
99.99 % uptime
Today
Newsletter emails   Operational
90 days ago
99.99 % uptime
Today
Web Interface Operational
Email reports and statistics   ? Degraded Performance
Activity logs   ? Operational
Backend Processing Operational
Imports and Exports   ? Operational
Segment calcluation   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
API Response Times
Fetching
Past Incidents
Oct 19, 2017

No incidents reported today.

Oct 17, 2017

No incidents reported.

Oct 16, 2017

No incidents reported.

Oct 15, 2017

No incidents reported.

Oct 14, 2017

No incidents reported.

Oct 13, 2017

No incidents reported.

Oct 12, 2017

No incidents reported.

Oct 11, 2017

No incidents reported.

Oct 10, 2017
Between 13:00 UTC and 15:18 UTC 9 October we observed a drop in our overall newsletter throughput.

This manifest in several outcomes:

• Some email campaigns in this window started sending late. These delays were typically in the order of 15-20 minutes.
• Email campaigns sending during this window sent at around 30% of the typical rate.
• Due to these delays, a portion of some email campaigns were blocked from sending. These campaigns entered a retry loop, meaning that the jobs retried over an increasing period – this meant that a small portion of some email campaigns (2-8%) were sent up to four hours late.

We have found the root cause of this issue. Ultimately a rogue process was killing one of our core email sending engines. We have put in place two changes:

1. We believe we have fixed the root cause by adjusting the way one of our email processors runs. This should ensure this particular root cause does not occur again.
2. Our alerting system did alert us to this problem, but took around an hour to do so. We have increased the aggressiveness of our monitoring over this component, meaning we should be alerted faster.

Apologies for the inconvenience caused.

If you have any questions, please email us at support@getvero.com.

Thanks,
Team Vero
Oct 10, 17:29 AEDT
Oct 9, 2017

No incidents reported.

Oct 8, 2017

No incidents reported.

Oct 7, 2017

No incidents reported.

Oct 6, 2017

No incidents reported.

Oct 5, 2017

No incidents reported.