Firstup Status - Platform Service Disruption - User Bulk Imports Failing

Platform Service Disruption - User Bulk Imports Failing - US Datacenter Only

Incident Report for Firstup

Postmortem

On Tuesday May 9th 2023, a configuration change was made in our Database (DB) environment to increase DB performance.

On Friday May 12th 2023, the above change surfaced an issue in our Bulk Upload system where queries would run for too long, resulting in DB locking. As a mitigation step, these locks were cleared allowing Bulk Upload jobs to continue processing successfully.

The root cause was later identified to be a missing DB index, and on Monday May 22nd 2023, the missing DB index was added to resolve the DB locking issue.

Posted Jun 12, 2023 - 19:44 UTC

Resolved

After extensive monitoring of the services that were affected by this service disruption, they all have remained available and operational.

This incident is now being placed in a resolved status.

Posted Jun 02, 2023 - 14:45 UTC

Monitoring

The proposed hot fix for this issue has been confirmed and approved, and has now been deployed in the production environment.

For any failed bulk import jobs, the files can now be re-uploaded for processing.

We will be placing this issue under monitoring for now.

Posted May 22, 2023 - 14:58 UTC

Identified

We have identified a potential root cause for this issue, and a hot fix for it is now being tested in a staging environment. Once confirmed and approved, it will be deployed in the production environment.

We will provide you with another update within 1 to 2 hours.

Posted May 22, 2023 - 14:40 UTC

Update

We continue to investigate this recurrence, and will provide another update within 1 to 2 hours.

Posted May 22, 2023 - 13:35 UTC

Investigating

We are investigating a recurrence of the issue affecting user imports. At this time, bulk user imports are failing with an 'Internal Server Error' for customers with communities in our US Data Center.

We will provide an update within 1 hour.

Posted May 22, 2023 - 11:08 UTC

Monitoring

User Bulk Import service is still running as expected.

We will place this service under monitoring, and provide an update if and when new information is made available.

Posted May 12, 2023 - 16:30 UTC

Identified

This issue has now been mitigated, and user bulk imports are now processing as expected.

We are working on backfilling any previously failed jobs.

An update will be provided once this is completed.

Posted May 12, 2023 - 14:01 UTC

Investigating

We are aware of, and are currently investigating an issue where bulk user upload jobs are failing with an internal server error in our US datacenter.

We will provide an update within 1 hour.

Posted May 12, 2023 - 13:20 UTC

This incident affected: Platforms (US Firstup Platform).