Web Experience and API are currently unavailable for EU Customers.
Incident Report for Firstup
Postmortem

Summary:

On Thursday, May 31st, 2024, we received reports from several EU communities that the Web and Mobile Experiences were malfunctioning. End users reported that the Web or Mobile Experiences weren’t loading as expected, and a blank screen was observed.

Scope:

The scope of this incident was restricted to end users trying to access the Web and Mobile Experiences for communities in the EU only.

Root Cause:

A platform incident was declared at 08:30 UTC, and the incident response team identified a misconfiguration on the EU instance of CloudFront cache policy. The misconfiguration made it so that the first request for an asset from any origin would be cached as a response to asset requests from all other origins. This caused subsequent requests from all other origins in the EU to fail due to Cross-origin resource sharing (CORS), resulting in the blank screen observed.

Mitigation:

The immediate impact was mitigated by invalidating the incorrect cached response in the EU instance of CloudFront, which restored Web and Mobile Experience access by 09:35 UTC. Additional assets-reindexing was performed to force any end-user mobile devices that had locally cached the incorrect responses to reload the Mobile Experience.

Recurrence Prevention:

The EU CloudFront cache policy configurations have been updated to match the US settings to prevent this incident from recurring.

Posted Jun 10, 2024 - 15:16 UTC

Resolved
This incident has been resolved.
Posted Jun 04, 2024 - 22:56 UTC
Monitoring
We have applied a fix for the issue affecting Web Experience and Partner API for EU communities and are currently monitoring.
Posted May 31, 2024 - 09:39 UTC
Identified
We have identified a possible root cause for the issue affecting Web Experience and the Partner API for EU customers and will provide further updates once we have applied a fix.
Posted May 31, 2024 - 09:09 UTC
Investigating
We are investigating a service disruption affecting Web Experience for EU customers. Our Engineering Team are urgently investigating. Updates to follow.
Posted May 31, 2024 - 08:42 UTC
This incident affected: Platforms (EU Firstup Platform), Products (Web Experience), and Ecosystem (Partner API).