Delay in segmentation of the condition "has triggered event with property"
Incident Report for Vero
Postmortem

We're consistently working to evolve Vero's underlying infrastructure to make the platform more efficient and secure.

At the end of March we made some changes to the core database powering "event property" conditions. As a result of this change a series of maintenance tasks failed between March 26th and March 31st. As a result, at March 31st at 4:27am UTC we began observing delays with newsletter and segment calculations that relied on the event data property condition.

We subsequently diagnosed the root cause of the database performance degradation and fixed and reinstated the maintenance operations. As a significant amount of data had accumulated during the intervening period we had to balance running new processing for segment calculations and the maintenance operations which needed to catch up on the data that had not been processed since the incident first occurred.

We saw some gradual improvements as the maintenance progressed and, since this process fully completed (April 11th, 2:35pm UTC), we have seen segment calculations return to nominal performance.

As part of the (unrelated) scheduled maintenance operations that also took place over the weekend of April 10th and 11th we were able to make changes to the processing system that will prevent this occurring again.

We are adding further monitoring to ensure any future issues would be detected more promptly.

If you have any questions, please contact us at support@getvero.com.

Posted Apr 14, 2022 - 16:01 AEST

Resolved
Segments with the condition "has triggered event with property" are now calculating in the expected time. This means any related newsletters will also be sending as expected.

We'll be following up with a detailed post mortem shortly. In the meantime please contact support@getvero.com if you have any questions related to your account.
Posted Apr 12, 2022 - 17:12 AEST
Monitoring
Maintenance operations to optimise segment calculation speed have completed over the weekend and we are monitoring the results.
Posted Apr 11, 2022 - 19:52 AEST
Update
Maintenance operations to optimise segment calculation speed are continuing to run. Additionally the system maintenance we will complete this weekend will contribute to improvements.

Only the condition "has triggered event with property" is affected with all other segment conditions running efficiently.

If you utilise this condition for a newsletter you may see some delay, we'll reach out if this is the case and can provide more information regarding your account.

We'll follow up on Monday with an update on the status of this issue and will ofcourse follow up with a detailed postmortem once this is resolved.

If you have any questions about your segments or newsletters please email support@getvero.com
Posted Apr 08, 2022 - 17:49 AEST
Update
We've identified the root cause of the performance degradation and we are running maintenance operations to resolve the issue. This maintenance operation will run for the remainder of the week. We will provide another update on progress within 24 hours.

In the meantime we've been taking mitigating actions to ensure that affected segments and newsletters continue to process, though you may still experience delays.

Please email support@getvero.com if you have any questions regarding your account, segments or newsletters.
Posted Apr 07, 2022 - 19:46 AEST
Identified
On Thursday 31st March at 4:27am UTC we began to experience delays in calculating segments that reference the condition "has triggered event with property". This is a result of a performance degradation on the database responsible for segmentation of events.

If your newsletter references a segment that queries for the condition "has triggered event with property" it may be delayed whilst we perform maintenance on this database.

We will provide further updates as soon as they become available.

Note that automated messages (Workflows, Behavioural and Transactional campaigns) remain **unaffected** and are **operating as normal**.

Please email support@getvero.com if you have any questions regarding your account, segments or newsletters.
Posted Apr 04, 2022 - 16:41 AEST
This incident affected: Vero Workflows: Segment calculation.