Summary:

On March 15th, 2024, we started receiving reports where scheduled campaigns experienced delays in publishing at the scheduled time or did not publish at all at the scheduled time.

Impact:

The impact was restricted to any scheduled campaigns on the FirstUp platform scheduled to publish on March 15th, 2024, between 1:00 AM ET (05:00 UTC) and 8:04 PM ET (March 16th, 2024 - 00:04 UTC).

Root Cause:

The root cause was determined to be a regression to a software change to the “scheduled campaign callback service” that was deployed during our scheduled software release window the previous day causing a callback to the “scheduling service” (to publish a scheduled campaign at the scheduled time) to fail.

Mitigation:

A hotfix was deployed by 8:04 PM ET (March 16th, 2024 - 00:04 UTC) to address the software regression introduced in the campaign scheduling software. Any delayed scheduled campaigns were also manually published by the same time.

Recurrence Prevention:

The Incident Response Team has taken the following actions in an effort to prevent a recurrence of this incident:

• Implemented additional pre-release regression testing around the “scheduling service”.
• Documented the SQL rake task used to identify any failed/delayed scheduled campaigns in a runbook to aid in quickly mitigating any future similar incidents.
• Created monitors to alert us on the first instance of a failed/delayed scheduled campaign to enable us to proactively get ahead of any campaign scheduling issue(s) and prevent similar platform-wide incidents.