From 15:00 UTC 24 March we observed that our send rate across all customers newsletters was approximately half of what it usually is.
Our team responded to this issue immediately by increasing the resources available to our newsletter queues as a temporary solution whilst we investigated the root cause. Unfortunately this did not lead to a full recover in speed, though did mean that newsletters continued to send at a reasonable rate.
Over the following five to six hours we ran a number of tests and discovered that the root cause stemmed from two things. The first was increased latency in requests made to one of our external data providers, the second was a series of low-level database locks that were related to ensuring users are sent email in the correct order and with the correct variation – these were operating in a non-performant manner.
From 15:00 UTC and 20:30 UTC in particular you would have experienced these scenarios:
By 21:00 UTC all email delivery had returned to normal. We have monitored this for a further ten hours to ensure things are operating completely normal before closing this issue.
Email delivery and email speed are the two key technical challenges related to getting your emails securely and reliably to your customers that matter at Vero. Necessarily, we have taken this very seriously. We appreciate your patience.
I'd like to finish with a little more context on this issue. Since November 2015 we have seen extremely consistent send speeds. We made a big upgrade at that time, and it's made Vero more usable for all our customers.
If there is any irony in last night's situation it is that this weekend was slated for the release of our next core upgrade to email delivery which should see send speeds increase by up to four times. Again, a huge win for everyone on the Vero platforms – particularly in light of the power of our external data and personalization tools.
We are proceeding today preparing for that launch and, all tests going well, will introduce it as planned over the next 24 to 48 hours. There should be no noticable difference other than your newsletters sending faster than ever before! We look forawrd to bringing this to you.
Have a great weekend and a great Easter.
James
--
CTO, Vero
jamesl@getvero.com