Platform Service Degradation - Email Delivery Inconsistencies

Incident Report for Firstup

Postmortem

Summary:

On June 22nd, 2026, we started receiving reports where email campaigns were being delivered to only a subset of the targeted email audience. Following initial triaging and troubleshooting, a platform incident was declared, and the incident response team began looking at the reported issue.  The unexpected behavior was determined to be due to a logic error in user reachability through the email channel, and the software change implicated was immediately rolled back to resolve the issue.

Severity:

Sev-2

Scope:

Some campaigns sent between 12:54 AM PT on June 20th, 2026, and 12:59 PM PT on June 22nd, 2026, to an email audience from the Firstup platform.

Impact:

Emails were not sent to the entire email audience for campaigns sent during the incident duration.

Root Cause:

The root cause for this incident was determined to be a regression from a software change that was made on June 20th, 2026, to align with the reachability definitions that we introduced in the Q2 quarterly release. A software bug was identified that returned a “Unknown” email address reachability status when an email audience was being processed for campaign delivery in the backend, resulting in those email addresses being excluded from the campaign’s delivery backend path.

Mitigation:

To immediately mitigate this incident, a rollback of the software change was performed, which was completed at 12:59 PM PT, restoring the reachability data to its previous state platform-wide. For customers who had campaigns in scope of this incident, and needed to send the email to users who were initially excluded from the campaign’s delivery, our Support team provided a list of those users to be retargeted separately via a duplicate campaign.

Recurrence Prevention:

Our recurrence prevention measures include:

  • Investigate and fix the software bug that returned the “Unknown” email address reachability status.
  • Update our QA processes to include robust end-to-end testing on all New Product Features and Enhancements before being released.
Posted Jul 01, 2026 - 16:38 UTC

Resolved

This incident is now resolved and all components marked fully operational. No further issues observed while monitoring since the platform incident was mitigated at 1PM PT today.
Posted Jun 23, 2026 - 01:41 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 22, 2026 - 20:09 UTC

Update

We have completed the code change rollback, and this incident is now mitigated. We will be placing the impacted component(s) under active monitoring to ensure continued proper functionality.
Posted Jun 22, 2026 - 20:09 UTC

Update

We currently estimate that the code change rollback will be completed within the next 2 hours, which will fully mitigate the impact of the observed behavior. We will provide another update as soon as this is completed.
Posted Jun 22, 2026 - 19:11 UTC

Update

We are still in the process of rolling back the code change we believe caused the observed behavior. We anticipate this should be completed soon, but we will provide another update on its progress within the next hour.
Posted Jun 22, 2026 - 18:51 UTC

Identified

We have identified a code change that resulted in the observed behavior. We are currently rolling back that code change and will provide another update once that is completed.
Posted Jun 22, 2026 - 17:48 UTC

Investigating

We are investigating reports where campaign emails are not being delivered to the entire targeted campaign audience.

We will provide you with an update within 1 hour.
Posted Jun 22, 2026 - 17:11 UTC
This incident affected: Products (Campaign Delivery).