Some images requiring 3rd party transformation, not rendering as expected on Studio and Web Experience

Summary:
On November 25th, customers experienced issues where images that required transformation (i.e., images with rounded corners, video preview images, inline images in rich text format) stopped displaying across various parts of the platform. It became evident that the failure was related to a third-party image processing partner inadvertently disabling a critical feature used by Firstup during an automated service renewal process.

All affected images began displaying normally once the configuration was fixed by the third-party provider.

Impact: 

• Transformed images failed to display for customers.
• No images were lost or corrupted; the issue solely affected their ability to be displayed.
• Service returned to normal after the third-party provider corrected the configuration and cleared relevant caches.

Root Cause:
An incorrect change in service entitlements was made during the account renewal process with the provider. This misconfiguration impacted image delivery capabilities resulting in the “broken image” icon being displayed rather than the desired image.

Mitigation:
An incident was declared at 9:12 am PST. Upon clarifying the issue's scope and origin, engineers first validated that no recent Firstup code changes were responsible. Subsequent testing of image URLs and delivery paths led to the engagement of our third-party provider for deeper investigation. The provider confirmed the root cause was a misconfiguration in the account and their engineers applied a correction and cleared caches for the production environment, which subsequently restored image functionality.

Recurrence Prevention: The following has been identified as follow-up to commit to as a part of this process: 

• Ensuring the necessary features are explicitly included in contract details during the contract negotiation process.