User Imports stuck in Created status

Summary:

On November 14, a large file uploaded by one customer program caused the system responsible for processing user imports to slow down significantly. Because the system processes import jobs in order, this slow-running file created a backlog that delayed other customers’ imports during the incident.

Once the issue was identified, the problematic file was stopped and the system was restored, allowing imports to resume normal processing.

Impact:

• CSV files uploaded during the affected timeframe experienced delays.
• No data loss occurred, and all delayed imports completed successfully once the system recovered.

Root Cause:

A large CSV file uploaded by one customer program contained a high volume of duplicate entries, which triggered recursive processing behavior. This caused the import system to spend an unusually long time processing that single file, slowing down the overall import queue.

Mitigation:

An incident was declared as soon as the scope was understood. After identifying the slow-running upload as the source of the delays, the team stopped the problematic job to prevent it from continuing to block the queue. Performance was then restored by restarting the processing workers and performing maintenance to improve lookup speeds. These steps allowed the system to clear the backlog and return to normal operation. All delayed import jobs completed successfully once the processing queue began moving again.

Recurrence Prevention:

The following actions have been identified as follow-up actions to commit to as a part of this process:

• Adding monitoring that alerts us if import jobs begin taking unusually long or if the queue grows too quickly.
• Improving how we validate uploaded user data.
• Enhancing our import processing logic so that slow-running jobs can be isolated without delaying other customers.