Web Experience is displaying a 'signed out..' error for some programs when trying to access.

Summary: 

On August 5th, 2025 we began receiving Customer reports of the following error being displayed by the Web Experience channel for impacted users: "You've been signed out or your access has been expired". This behavior was attributed to unexpected system memory pressure which resulted in resource starvation on the platform and subsequent intermittent user session invalidation.  The initial attempt to mitigate impact by doubling the size of the system cache worked temporarily.  However, further action was needed to quarantine a recently published campaign determined to be the cause of the out-of-memory condition. This secondary action fully restored service for the Web Experience channel. 

‌

Impact: 

A system error message was displayed intermittently to end-users accessing the Web Experience between 08:21 AM - 09:01 AM PT and again between 11:23 AM - 11:37 AM PT.

"'You've been signed out or your access has been expired" 

Other services and endpoints including Creator Studio and the Mobile Experience remained fully functional. 

‌

​Root Cause: 

Root cause was determined to be data store instances being above memory capacity resulting in intermittent user session invalidation. This behavior was attributed to unexpected increased memory utilization due to several factors:

1. Large number of communications/campaigns being sent at the same time near US peak hours which increased cache-pressure.
2. Non-optimal caching of a recently published Classic Studio campaign containing multiple large images in the campaign content body itself.
3. Increase in ‘401 unauthorized’ errors due to a newly discovered software bug which directly led to the "You've been signed out or your access has been expired" error message on the front end.

‌

Mitigation: 

A platform incident was declared at 08:31 am PT on Tuesday, August 5th and at 08:41 AM PT the incident team determined the root cause to be related to memory exhaustion. Issue was attempted to be mitigated earlier by and faster by doubling the size of the cache(data store) instance which temporarily restored service.  The incident recurred approximately 2 hours later, and further analysis revealed non-optimal storage-reuse (ie caching) behavior related to a recently published campaign.  That implicated campaign was quarantined to address the immediate impact, thus fully restoring service for the Web Experience channel.

‌

Recurrence Prevention: 

The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA (Root Cause Assessment) process: 

• Campaign body caching has been optimized so we will only cache the data that is being used in the response.  This change has already been completed, and we have observed significant improvement in cache optimization by both reducing the size of allowed cacheable objects and also caching the same object for multiple users.
• Additional real-time monitoring has been implemented to improve threshold warnings for high memory usage.
• We have implemented a hotfix for the behavior causing the user session invalidation error message so that all cache-calls go through new logic that fall back to the source of truth and are resilient to Data Store cache issues. This has put us in a state where we will not have a re-occurrence of the unintended user-facing error message, and will drastically reduce the chance of using up the full memory.
• Testing and Analysis - Rigorous load testing on non-production internal environments was performed to verify expected behavior for campaigns with large audiences containing imagines inline with the content body.