Summary:

On Thursday, May 31st, 2024, we received reports from several EU communities that the Web and Mobile Experiences were malfunctioning. End users reported that the Web or Mobile Experiences weren’t loading as expected, and a blank screen was observed.

Scope:

The scope of this incident was restricted to end users trying to access the Web and Mobile Experiences for communities in the EU only.

Root Cause:

A platform incident was declared at 08:30 UTC, and the incident response team identified a misconfiguration on the EU instance of CloudFront cache policy. The misconfiguration made it so that the first request for an asset from any origin would be cached as a response to asset requests from all other origins. This caused subsequent requests from all other origins in the EU to fail due to Cross-origin resource sharing (CORS), resulting in the blank screen observed.

Mitigation:

The immediate impact was mitigated by invalidating the incorrect cached response in the EU instance of CloudFront, which restored Web and Mobile Experience access by 09:35 UTC. Additional assets-reindexing was performed to force any end-user mobile devices that had locally cached the incorrect responses to reload the Mobile Experience.

Recurrence Prevention:

The EU CloudFront cache policy configurations have been updated to match the US settings to prevent this incident from recurring.