Logs delayed and unavailable
Incident Report for Vero
Resolved
This incident has been resolved.
Posted Sep 23, 2019 - 14:28 AEST
Update
All Event logs were restored at the end of last week though we will continue to monitor for any further issues.
Please email support@getvero.com if you have any questions.
Posted Sep 23, 2019 - 14:15 AEST
Update
Event logs are now backfilled to September 4th and new events continue to be indexed in realtime.

Please email support@getvero.com if you have any questions regarding access to event logs older than September 4th and we'd be happy to assist.
Posted Sep 18, 2019 - 19:13 AEST
Update
Event logs (for up to the last 24 hours) have now been restored and are being indexed (displayed) in real time 🎉. You should now be able to search for events in the UI, access event logs and view event data for your customers.

We will continue to monitor and are working hard to restore all logs and maintain stability moving forward.

Please email support@getvero.com if you have any questions regarding your account.
Posted Sep 17, 2019 - 04:00 AEST
Update
Access to Logs, Customers and Customer Profiles continues to be stable. We are processing all Email logs and User updates in real time.

After assessing the root issue and the related down time, we made the decision on Friday evening PT to upgrade one of our search clusters as part of the data restore. This will save future planned downtime and ensure greater stability.

Our current aim is to be processing real time Event logs by Wednesday morning PT or earlier. We aim to have the full Event log history restored by the end of this week. We will provide further updates as we proceed.

We again apologise for the inconvenience. Processing of Emails and User updates has NOT been delayed or affected by this incident – it has affected the UI only. If you have questions, concerns or would like us to access data on your behalf, please email us at support@getvero.com.
Posted Sep 16, 2019 - 10:59 AEST
Monitoring
Access to the Logs, Customers and Customer Profile pages has now been restored. This means you are now able to load the customers page, search for customers and access their profile pages.

At this time, the Logs and Customer Profile pages include all *email* logs since approximately 11:30pm UTC Friday 13 September.

*Event* logs are currently unavailable. We are currently working on restoring these logs so they are visible.

We want to clarify that this has *NOT* affected any processing, solely the UI display of our logs records. Please contact support@getvero.com if you have any question about your account or access to user data.
Posted Sep 14, 2019 - 10:36 AEST
Identified
Over the last 12 hours we have experience performance degradation in the the search cluster powering the Vero Logs, Customers and Customer Profile pages. This has led to delays processing new data into the log search, as well as the inability to access these pages from the UI. We have noted this using the Components below.

We have now identified the cause. In order to restore normal, full access we need to perform a restore from a cold backup. We anticipate that within the next couple of hours we will have restored normal availability for:

• The Customers page.
• Email logs on the Logs and Customer Profile pages.

We will provide further updates regarding logs related to event tracking shortly.
Posted Sep 14, 2019 - 06:58 AEST
This incident affected: Vero Cloud: UI (Logs page activity).