On Wednesday May 15th, 2024 starting at approximately 3:00 AM PT until 10:22 AM PT we received customer reports that scheduled campaigns were not being delivered as expected. These reports included campaigns either sitting in the ‘Scheduled’ queue and not getting sent on time, as well as reports of campaigns that were marked as ‘Delivered’ but showed no actual sends to the users in the Campaign Delivery Report. The problem was linked to a regression in a service used for scheduling that introduced a new post related variable that was not correctly initialized in a specific campaign. The result was that this campaign, and all subsequently scheduled campaigns, were not automatically published during the affected time.
Campaigns that were scheduled remained sitting in the scheduled queue, or were incorrectly marked as delivered, but were not actually sent to the target audience. We confirmed at least 50 campaigns that were scheduled to publish experienced some sort of a delay.
The latest deployment of the service used for campaign scheduling had an internal permissions check failure that resulted in an elevated error rate as a result of a missing expected default value that was not set.
Once the root cause was identified through a source code analyzer, we corrected the post with the missing expected default value. This allowed all of the remaining scheduled campaigns to be published. Any other campaigns that did not go out on time were then manually published by the incident management team.
The below changes have been implemented to ensure campaign delays are prevented due to internal deployment activities: