Summary:
On Tuesday, January 21st, 2025, starting at around 12:35 PM PT, we started receiving reports that the Employee Experience was not visible, but rather, a “Under Construction” notice displayed in its place. Users were unable to close the notice and proceed to the Employee Experience. Initial investigations indicated that the reported issue was further reaching than just the customers who had reported it, and a platform incident was declared and published on the Firstup status page at 12:58 PM PT.
Severity:
Sev2
Scope:
The scope of this service disruption included all end users on the Firstup platform who attempted to access the Employee Experience in the duration of this incident.
Impact:
In the duration of this incident (30mins), end users could successfully log into the Employee Experience but saw a “Under Construction” notice instead of their expected Employee Experience frontpage and could not navigate away from the notice overlay. The Employee Experience was still available behind the Pendo guide overlay, and no data was compromised as a result of the incident.
Root Cause:
The incident response team quickly identified the root cause of the incident to be a Pendo guide overlay that was erroneously published by a Firstup employee across the entire Employee Experience platform at 12:29 PM PT. This guide was intended for a specific customer account, which was not correctly tagged during its publication, and therefore, visible to our entire customer base.
Pendo is a third party tool that allows for in-app annotations and user messaging that is primarily used by the Product Management and Marketing organizations primarily targeting program managers actively using Creator Studio.
Mitigation:
To resolve this incident, the Pendo guide was deactivated at 12:59 PM PT, which restored visibility to the Employee Experience immediately.
Recurrence Prevention:
The following actions have been identified as follow-up items to prevent a recurrence of this incident: