Partially Degraded Service
Vero Cloud: Ingestion API ? Operational
90 days ago
100.0 % uptime
Today
Vero Cloud: Newsletter processing ? Operational
90 days ago
100.0 % uptime
Today
Vero Cloud: Automated email processing ? Degraded Performance
90 days ago
100.0 % uptime
Today
Transactional emails Operational
90 days ago
100.0 % uptime
Today
Behavioral emails Operational
90 days ago
100.0 % uptime
Today
Workflows Degraded Performance
90 days ago
100.0 % uptime
Today
Vero Cloud: Segment calculation ? Operational
90 days ago
100.0 % uptime
Today
Vero Cloud: Reports data availability ? Operational
Reports data availability (Vero default and Mailgun integrations) ? Operational
Reports data availability (Sendgrid and other non-Mailgun integrations) ? Operational
Vero Cloud: UI ? Operational
General UI access and speed ? Operational
Logs page activity ? Operational
CSV Imports and Exports ? Operational
Vero Connect: Newsletter processing ? Operational
90 days ago
100.0 % uptime
Today
Vero Connect: Reports data availability ? Operational
90 days ago
100.0 % uptime
Today
Vero Connect: UI ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Apr 13, 2024

No incidents reported today.

Apr 12, 2024

No incidents reported.

Apr 11, 2024

No incidents reported.

Apr 10, 2024

No incidents reported.

Apr 9, 2024

No incidents reported.

Apr 8, 2024

No incidents reported.

Apr 7, 2024

No incidents reported.

Apr 6, 2024

No incidents reported.

Apr 5, 2024

No incidents reported.

Apr 4, 2024

No incidents reported.

Apr 3, 2024

No incidents reported.

Apr 2, 2024

No incidents reported.

Apr 1, 2024
Resolved - At 11:17 UTC (1 April), our team was alerted to delays in automated campaign processing. At this time we marked behavioural, transactional and workflow processing as degraded. This affected all customers.

By 11:54 UTC, our team had isolated the root cause: delayed filter evaluations on one of our data sources.

At 12:30 UTC, processing performance had returned to normal but there was a large backlog of automated campaign jobs to work through. We continued to monitor as the backlog reduced and began work on a fix to the root cause.

At 18:49 UTC, we noticed a spike in processing again. This was quickly resolved (as of 19:30). We continued to monitor the backlog.

As of 21:30 UTC, the backlog has cleared and workflow processing is 100% realtime for all but handful of customers (with larger backlogs). These remaining backlogs should be fully cleared by 00:00 UTC.

At 21:56 we deployed a patch for the root cause. We believe we have fixed the underlying cause and this will not occur again. We will monitor extra closely over the coming 48 hours to ensure this is the case.

If you have any questions, please email support@getvero.com. Thank you.

We also want to note that this incident was unrelated to the database upgrades/maintenance conducted over the weekend. These changes have gone smoothly.

--

Processing has returned to normal and we can report the following metrics for the key time windows. These are the P50 to P90 reports for workflow node processing. This represents how long it takes to process a node vs. when the node was queued for evaluation. We expect a p90 of 5 minutes across all nodes. Note that this include **all node types** across all workflow types (transactional, non-transactional, etc.). For example, A node with 50 conditions looking back across a large date range will take longer to evaluate than a simple email node.

10:00 - 21:30 UTC 1 April (catching up on backlog)
• p50 = 1 hour
• p75 = 2.45 hours
• p90 = 4.6 hours

21:30 - 00:00 UTC 1 April (tail of backlog, with majority of customers in realtime)
• p50 = 48 minutes
• p75 = 1.8 hours
• p90 = 3.1 hours

00:00 - 01:00 UTC 2 April (backlog fully processed and processing in realtime)
• p50 = 31 seconds
• p75 = 3 minutes
• p90 = 5.4 minutes

Apr 1, 22:30 AEDT
Completed - The scheduled maintenance has been completed.
Apr 1, 12:12 AEDT
Update - We are continuing to verify the maintenance items.
Apr 1, 11:55 AEDT
Verifying - Verification is currently underway for the maintenance items.
Apr 1, 11:41 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 1, 11:00 AEDT
Scheduled - We will be performing system upgrades on some of our databases, commencing Monday 1 April at 00:00 UTC (Sun 5pm PDT / 8pm EDT, Mon 1am BST / 2am CEST / 11am AEDT)

The aim of this maintenance is to improve system performance and security.

There will be some disruptions across services including delays to API and email processing for approximately 30-45 minutes during the maintenance period. There will be no data loss during the downtime and all pending API requests will be processed once maintenance is complete.

If you have any questions or concerns, please email us at support@getvero.com.

Mar 29, 10:36 AEDT
Mar 31, 2024

No incidents reported.

Mar 30, 2024

No incidents reported.