Service Disruption - Studio and Experience

Incident Report for Firstup

Postmortem

The root cause of this service disruption has been identified to be the same as the one posted in another incident’s postmortem. Please follow this link for additional details.

Posted Feb 29, 2024 - 19:36 UTC

Resolved

This incident has been fully resolved and all components remain fully operational.
Posted Jan 30, 2024 - 16:02 UTC

Monitoring

This issue has now been fully mitigated and all components and platform are fully operational. Maintaining in a monitoring state while remaining checks are performed.
Posted Jan 25, 2024 - 18:58 UTC

Identified

We are seeing incremental improvements in error rates platform wide and improved responsiveness.
Posted Jan 25, 2024 - 18:14 UTC

Update

We are restarting incremental services that have been determined to be holding on to network connections we feel may be leading to an unhealthy state.  As a result some observed symptoms may change or worsen for a short period of time.
Posted Jan 25, 2024 - 17:48 UTC

Update

We are now working in close contact with our Platform vendor AWS, to identify the source of elevated database locks that are causing inconsistent service performance and availability across multiple endpoints.
Posted Jan 25, 2024 - 17:14 UTC

Update

We are continuing to investigate the service disruption affecting Studio and Experience.

We'll update this incident as soon as we have further details.
Posted Jan 25, 2024 - 16:55 UTC

Investigating

We experienced an unanticipated service disruption across all components. This may have affected Studio and the member experience. Users may have seen an error page, or experienced slowness when accessing the platform.

We are investigating this issue and will provide an update in due course. Please monitor this page for further updates.
Posted Jan 25, 2024 - 15:13 UTC
This incident affected: Platforms (US Firstup Platform).