Email Deliverability Service Disruption

Summary:

Shortly after 11:30AM PT on December 19th, 2024 Firstup began to receive Customer reports of emails not being received. An incident was declared an hour later after it was determined Proofpoint’s dynamic reputation engine had flagged all three Firstup production sending IPs, resulting in many emails being blocked by the recipient email server(s). This observation was shared back with Customers impacted so that they could request that IPs be removed from Proofpoint’s list. Firstup also requested the same through Proofpoint’s public facing page. The IPs were confirmed to have been removed from the blocklist by 1:52PM PT which immediately mitigated impact from further emails being negatively affected.

Impact:

All email in scope of the incident was not delivered to the intended recipients, nor was there an attempt to redeliver the original emails due to the nature of the error Proofpoint email servers sent back to Firstup. Proofpoint was responding to all emails from these IPs with the following error:

• 554 Blocked

No users were unsubscribed as a result of the delivery error, so any email sent to the same users after the sending IPs were removed from Proofpoint’s blocklist were successfully received. However, any emails that were blocked and not sent using optimized delivery with retargeting, had to be resent through Creator Studio as they were not quarantined or deferred by the email systems. Longer running campaigns configured to include retargeting, would have continued to function beyond the day of the incident and likely resent emails that were originally blocked.

Root Cause:

Root cause was determined to be related to Proofpoint’s Dynamic IP Reputation Engine. There are a small number of triggers that Proofpoint discloses publicly in their FAQ (https://www.proofpoint.com/us/support-services/ip-blocked-faq) that Firstup has confirmed did not occur. Because the inner workings of this system are proprietary, and Proofpoint has not provided any specific remediation steps for tickets opened to remove the IPs from the list, our conclusion is that a false positive indicator on this system misidentified the sending IPs. Firstup sends hundreds of thousands of employer to employee emails per hour to Proofpoint servers on any given day, and December 19th was no different.

Mitigation:

Impact was mitigated through a multi-pronged approach. Firstup used Proofpoint’s public-facing removal request system at https://ipcheck.proofpoint.com/ and also worked through Customer IT and email security contacts to request the removal using Proofpoint’s expedited removal request system. One or both of these measures resulted in the three affected sending IP addresses being removed within the hour after the requests were made.

Recurrence Prevention:

The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA (Root Cause Assessment) process:

• Customers affected by the incident have been encouraged to open their own ticket with Proofpoint, as well as confirm that the three sending IP addresses have been allowlisted on their own Proofpoint instance and have a higher priority than the PDR blocklist. Further information on allowlisting email can be found in the following Knowledge Article - https://support.firstup.io/hc/en-us/articles/4417455533975-Allowlist-Emails-from-Firstup
• More granular monitoring is being put into place to identify patterns of throttling, errors, and delivery failures by program ID, and an outreach program will be established to work with Customers that appear to be misconfigured, or whose email systems cannot handle the volume of email they are attempting to send at any given time.