Summary:
On September 4th, 2024, starting at 4:27 AM EDT, reports of Studio users unable to view or edit campaigns in Studio were received. Following a correlation of customer reports and initial troubleshooting, a platform service degradation incident was declared at 9:21 AM EDT, and published on our Status Page at 9:41 AM EDT.
Scope:
The scope of this service degradation was restricted to Studio users with multiple audiences assigned to them.
Impact:
Studio users who had multiple audiences assigned to them were unable to view or edit campaigns during the duration of this incident (12hrs 46mins). No scheduled campaigns were affected, and campaign viewing, editing, and publishing processes were not inhibited for other users with no audiences assigned or had a single audience assigned.
Root Cause:
The root cause of this incident was determined to be a regression to a misconfigured platform enhancement policy change intended to improve the efficiency of how user-assigned audiences were queried, which had been released at 12:07 AM EDT as part of the scheduled software release maintenance the same day.
Mitigation:
A rollback of the offending policy change was performed and completed by 12:53 PM EDT to restore access to Studio campaigns.
Recurrence Prevention:
To prevent this incident from recurring, we will perform the following actions before releasing the platform enhancement policy change in the future: