From 07:00 to 13:40 on 31 October, newsletters experienced delays.
These delays were due to the outage of one of our newsletter processing clusters. Our team had to manually intervene and, after investigation, was able to restore the cluster relatively quickly. Due to the volume of emails we process, unfortunately the backlog that was created during this minor outage caused delays of varying magnitudes over the following several hours.
The worst affected customers saw slow sends for two to three hours.
This infrastructure issue is not one we have seen before and appears to have been hardware-related. From what we have been able to ascertain, this issue is isolated and should not arise again – it was not driven by a systemic bug in our framework or application code. Further to our investigation, we have added specific reporting over the root cause to enable our systems to respond in minutes if this reoccurs – ensuring we head off any lasting impact before it begins.
If you have questions or concerns please email us at support@getvero.com – we're happy to share details. As ever we treat any such issue with great care and use it as an opportunity to make our system more resilient and fault-tolerant in the future.