Platform Service Disruption - Web Experience Degradation

Incident Report for Firstup

Postmortem

On Thursday, March 16th 2023, at 4:11 PM UTC, one of our database clusters started receiving an influx of unanticipated link safety verification requests from an external email client. This resulted in the database cluster being intermittently unresponsive, manifesting to the end user as the Web Experience being sluggish or not loading in some cases.

At 5:14 PM UTC, the unanticipated database connections were terminated, in effect restoring Web Experience functionality.

We have also deployed measures to appropriately handle future link safety verification requests, in order to prevent a recurrence of this issue.

Posted Jun 27, 2023 - 19:56 UTC

Resolved

All affected services have remained available and stable.

This service disruption is now resolved.

Posted Mar 16, 2023 - 20:57 UTC

Monitoring

This service disruption has now been mitigated, and all services are now available. We are working to identify the root cause of this issue.

We will provide another update as more information is made available.

Posted Mar 16, 2023 - 17:18 UTC

Identified

We have identified a database issue, and are working towards a fix.

Next update in 30 minutes.

Posted Mar 16, 2023 - 17:08 UTC

Update

Update: known symptoms include:

- Studio displaying error pages, or errors within individual components (e.g, Insights)
- Member Experience spinning, or individual components not displaying (e.g, Shortcuts not loading, errors showing in Assistant)

Another update in 30 minutes.

Posted Mar 16, 2023 - 16:51 UTC

Update

Update: Web Experience, Studio, and Shortcuts appear to be affected by this service disruption.

Another update in 30 minutes.

Posted Mar 16, 2023 - 16:43 UTC

Investigating

We are aware of, and are currently investigating an issue where Web Experience is not loading.

We will provide another update in 30 minutes.

Posted Mar 16, 2023 - 16:38 UTC

This incident affected: Products (Creator Studio, Web Experience).