Firstup Status - Platform Degradation - Multiple Services Affected

Platform Degradation - Multiple Services Affected

Incident Report for Firstup

Postmortem

On May 22nd 2023, one of our caching servers received an unexpected amount of load. This caused the depending services to be impacted, cascading to other services as well.

As mitigation step, we added a circuit breaker in order to limit the scope of impact on dependent services, as well as increased the memory in our caching server to promptly restore services.

As a long term preventative measure, we have increased the time-to-live for cashed data, which reduces the frequency of calls to the caching server, hence reducing the load on the server.

Posted Jun 30, 2023 - 19:52 UTC

Resolved

All services affected by this service degradation have remained available and fully functional.

This is now considered resolved.

Posted Jun 26, 2023 - 18:35 UTC

Monitoring

All database connections have been restored, and all services are now available and accessible.

We will be placing this incident under monitoring

Posted May 22, 2023 - 17:54 UTC

Update

As we continue to work on restoring all database connections, some services such as Web Experience are back up and running.

Another update will be provided within 30 minutes.

Posted May 22, 2023 - 17:31 UTC

Identified

A database connection issue has been identified as the cause of the multiple services being degraded. We are working to mitigate this issue.

Another update will be provided within 30 minutes.

Posted May 22, 2023 - 17:01 UTC

Investigating

We are investigating reports of multiple services being degraded. These may include - among others - Microapps and Web Experience.

We will provide an update within 30 minutes.

Posted May 22, 2023 - 16:35 UTC

This incident affected: Platforms (US Firstup Platform) and Products (Web Experience, Microapps).