Delays in segment calculation
Incident Report for Vero
Resolved
Hi all, we're going to resolve this incident at this time.

For the last 10 hours we have observed our segment queries operating at their regular speeds. The changes we made worked effectively and rapidly.

We will continue to monitor proactively for the coming 24 hours.

This issue was hard to detect and, having found the cause, we have learnt from this case. In response to this incident we are setting up further monitoring to help alert us to another incident of this nature ahead of time.

Again, we apologise for the inconvenience causes. Have a great day.
Posted Sep 15, 2017 - 06:56 AEST
Identified
We have been investigating this issue over the past several hours.

We have not provided updates as we have only recently had concrete leads in this investigation.

Firstly, to clarify, this issue has not affected **every** segment. This has made our investigation difficult, as our core metrics suggest that things are operating effectively. After much work, we've isolated that the segments affected are any segments that contain the "user has property" condition.

Over the last 72 hours, the calculation time on these segments has been steadily increasing and has now reached an average calculation time of approximately four times the normal average. There are also significant outliers leading to long delays for some customers.

We believe we've found the root cause and are in the process of correcting this. We estimate it may take 12+ hours for these segments to be back to their normal calculation times.

In the interim, if you are sending **large** newsletters that use segments with these conditions we recommend that you:

• Refresh the segments manually yourself prior to sending.
• Email us at support@getvero.com and let us know which campaign this is / these are.
• We will update our system so that the send goes out on time, as the segments have been pre-calculated.

We will provide another update once we start seeing good improvements in calculation speed. This may take several hours. We will update earlier if we have more information of course. We're going to leave this incident open until full resolution.

Thank you.
Posted Sep 14, 2017 - 14:00 AEST
Investigating
We are currently experiencing delays in calculating some segments. We're investigating the root cause for this at this time.

Segments are still calculating, but are taking significantly longer to do so. For some customers, this will cause delays in newsletter sends as newsletters force recalculations on segments.

We will keep you updated here. We have not previously seen such an acute occurrence of this issue and have added a new component above to reflect Vero's current status. Thanks.
Posted Sep 14, 2017 - 07:24 AEST