We are continuing to monitor for any further issues.
Posted Jun 29, 2021 - 08:55 AEST
Update
We are continuing to monitor for any further issues.
Posted Jun 28, 2021 - 20:39 AEST
Update
Over the past 12 hours we have monitored our systems.
As a result we know that there is data missing for user properties updated on Sunday 27 June UTC and are currently restoring the user property data from that period. This only affects segmentation as it relates to sending newsletters. It does NOT affect content. User property updates made **before** ~00:00 Sunday 27 June are complete and accurate. User property updates made **since** ~14:00 UTC Sunday 27 June are also complete and accurate.
If you update user properties regularly (e.g. daily) then this is likely to have minimal or no impact on your newsletters. If you imported specific data on Sunday 27 June UTC and are using it for segmentation in a newsletter then this data will not yet be available. If you have any concerns we recommend you hold off scheduling newsletters and/or reach out to us. We're happy to discuss options via support@getvero.com.
This update may take 24 hours or more. We will leave this incident up until the import is complete.
**Important:** Workflows use a different system for loading user properties and all user property filters are complete and accurate in Workflow processing. This only impacts user properties updated on Sunday 27 June for use in segmentation for newsletters.
Thanks again for your patience.
Posted Jun 28, 2021 - 11:06 AEST
Update
We have now fully restored the affected data store.
Newsletter processing has been reenabled. Newsletters that were affected over the last several hours will now be processed as we push through the backlog. We anticipate there will be some delays whilst this occurs.
There will be some ongoing work as we do further background syncing and monitor to ensure everything is returned to a stable state. We'll post a further update when we are comfortable this is the case, at which point we'll resolve this incident.
Thank you for your patience. If you have any questions, please email us via support@getvero.com.
Posted Jun 28, 2021 - 00:21 AEST
Update
We have made progress with our service provider and are currently restoring the affected data store.
We estimate this restore will take another ~3 hours. We will provide an update when the restore is complete and newsletters are processing as normal. We have emailed affected customers.
We also want to confirm that API tracking and Workflow processing have remained unaffected and are processing normally.
Posted Jun 27, 2021 - 20:26 AEST
Monitoring
We have been alerted that one of the databases powering segment calculation has failed.
This has affected a small portion of customers over the last six hours. This is an issue with our service provider.
We are working with our service provider to restore service and will post an update when that occurs. We will also investigate why we were not alerted earlier.