On February 12th, 2024, starting at around 9:20 AM ET, we received reports that some users were experiencing issues accessing Creator Studio. The reports included general slowness loading Studio functions, as well as system error messages such as “Failed to fetch. Try again” or “504 Bad Gateway”.

During our investigations, it was identified that the number of available database connections was exhausted, and therefore new Creator Studio requests could not establish a connection to the database.

To mitigate this situation, a rolling redeployment of dependent backend services that had connections to the database was performed to free up any “stuck” connections, thus allowing new Creator Studio requests to connect to the database.

As a recurrence prevention measure to this incident, we are working on reducing the demand for database connections in the short term via methods such as:

• Connection pooling
• Read/write splitting

‌

In the long term, we will be looking at:

• Upgrading our database infrastructure
• Increasing our database connection limits.