Summary:

On Wednesday, January 1st, 2025, starting at around 6:41 AM PT, we received reports that some campaign emails were not being delivered as expected. While some campaign emails were being delivered successfully, others were reported as being quarantined for customers with strict email security policies. A platform incident was declared and published at 7:50 AM PT, and an incident response team began investigating these reports immediately.

Severity:

Sev2

Scope:

The scope of this service degradation was restricted to Firstup customers with strict email security policies that did not allow email deliveries originating from IP addresses other than the already allow-listed Firstup dedicated sender IP addresses. Other campaign delivery channels, such as Push Notifications and the Assistant, were not affected by this incident.

Impact:

In the duration of this incident (16hrs 46mins), customers with strict email security policies may have observed that some campaign emails were being silently dropped, quarantined, or bounced because they originated from IP addresses other than the already allow-listed Firstup dedicated sender IP addresses.

Root Cause:

The root cause was determined to be a misconfiguration on Firstup’s account by our third-party email delivery vendor, where, on December 31st, 2024 at 4:38 PM PT, a Technical Account Manager (TAM) on the vendor’s side erroneously placed Firstup’s dedicated email sender IP addresses in a “warming” state. This reduced the number of emails each dedicated IP address could send daily. Once that threshold was reached on January 1st, 2025, the remaining campaign emails were sent out from other IP addresses that were not associated with Firstup, which triggered a security response from customers with strict email security policies (as designed and expected), resulting in those emails either being silently dropped, quarantined, or bounced.

Mitigation:

To resolve this incident, Firstup’s dedicated sender IP addresses were removed from the “warming” state at 9:24 AM PT, which restored their full capabilities and normal throughput.

Recurrence Prevention:

The following actions have been taken or committed to by our third-party email delivery vendor:

• Have set up coaching sessions with the TAM team, to ensure better understanding of this type of procedure.
• Confirmed that the TAM in question will not be on Firstup’s account again.
• General reinforcement on account change procedures will be provided to all our TAMs to ensure this is followed up properly.