Resolved -
This incident is now resolved and all affected components marked fully operational.
Aug 14, 02:31 UTC
Update -
Backfill for 08/13 complete and we are now back at normal operating cadence for new data being ingested. Dashboards are all up to date, and campaign delivery reports, and hoverover reports from the campaign results page are in the process of being refreshed.
Aug 14, 00:38 UTC
Update -
Backfill process continuing to run for 08/13 events. Blocker has been removed that required multiple restarted and we have more confidence now in the throughput of the backfill process that will be complete for 08/13 within the hour. Focus will then shift to historic events on 08/10 and 08/11 which are also in scope.
Aug 13, 22:41 UTC
Update -
All processing for events that occurred on 8/12 has concluded. There are events on 8/13 that are still being processed as that backfill is competing against new events coming in today. As a result, the backfill processing for today's events has had to be restarted several times-- adding unexpected delay to the completion time. Next update will be 1 hour from now, or following the next milestone completion, whichever is sooner.
Aug 13, 21:39 UTC
Update -
There have been a few unexpected slowdowns in the backfill processing and we now expect the completion to be at least another hour from now. Also, not originally considered in scope, Poll reporting is also affected and being backfilled along with other reporting data.
Aug 13, 20:15 UTC
Monitoring -
We are continuing to actively monitor the data backfill progress and remain on target for full restoration of normal processing times by 1PM PT. Only programs on the US1 region are in scope of the issue.
Aug 13, 18:59 UTC
Identified -
The issue has been identified that has caused a backup of data pipeline processing and are in the process of working through the data backfill. We expect reporting data to be fully caught up within the next 2 hours with incremental improvements already visible now.
Aug 13, 17:59 UTC
Investigating -
We are currently investigating a delay in data populating in Insights reports. Reports in scope include the Campaign Results Page, Campaign Performance, Videos overview, hover popup modal, Campaign Delivery, and possibly others.
Aug 13, 17:06 UTC