WorkOS: Elevated API Errors

Customer impact

Customer impact lasted from 22:40 to 23:03 UTC. Across our public edge, 24.1% of recorded requests returned server errors, and 17.2% ended with the client disconnecting. The table breaks down request groups by product.

Latency Samples

• Authenticate POST requests: At the peak, one-minute median latency reached approximately 32 seconds and p95 latency reached approximately 45 seconds, compared with roughly 65 ms and 140 ms before the incident.
• Directory Sync: Peak one-minute p95 latency for selected SCIM user-creation and PATCH-update operations reached approximately 22 seconds, compared with roughly 65 ms before the incident.

Root cause and contributing factors

Around 22:40 UTC, the Redis primary became unavailable to application clients. AWS ElastiCache promoted its replica, completing the failover at 22:40:17 UTC. New connections could reach the promoted node, but some existing application connections remained stalled.

Three application behaviors prolonged and amplified the interruption:

1. Flag-call timeouts did not recover stalled connections. An application-level deadline returned a fallback value after two seconds, but it did not cancel the underlying Redis command or replace the connection. Additional reads accumulated while clients waited to detect the connection failure. Some commands remained pending for approximately 16 minutes.
2. External waits held database connections open. Some flag checks occurred inside database transactions. Repeated waits of several seconds kept pooled database connections occupied and, in some cases, held row locks. Available connections were consumed, causing other requests to queue and fail. Database compute capacity was not the limiting factor.
3. A fallback selected an incompatible decryption path. A flag used during a decryption-service migration defaulted to the old behavior when flag evaluation failed. That sent some encrypted values to a legacy service that could not decrypt them, causing additional authentication failures.

The local cache reduced normal Redis traffic, but expired entries still depended on Redis. Our two-second deadline limited how long a flag caller waited; it did not provide prompt recovery of a stale, established connection. This was a gap in our resiliency safeguards.

Recovery and communication

Responders restarted affected application instances, which triggered reconnection to the healthy Redis instance. As connections recovered and queued work drained, error rates and latency returned to normal.

We also identified a notification gap: the initial status page update did not select the affected service components. Customers subscribed only to those components therefore did not receive the incident notification.

Timeline

All times are UTC.

• September 18, 22:40 — Customer-facing errors and latency increase. AWS completes promotion of the Redis replica at 22:40:17.
• 22:44 — Engineers declare an incident following automated alerts.
• 22:49 — The public status-page incident is opened.
• 22:53 — Investigation identifies stalled feature-flag Redis clients and database connection starvation.
• 22:55–23:03 — Responders restart affected application instances, existing clients begin reconnecting, and queued work drains. Recovery progresses across services.
• 23:03 — Customer-facing error rates and latency return to normal.
• 23:29 — The status-page incident is marked resolved after monitoring.
• September 19, 01:44 — A Redis-client recovery fix is deployed to production after validation with controlled failure and failover tests in staging.

Remediation

Immediate Fixes

Completed:

• Restored service: Restarted affected application instances and monitored recovery as clients reconnected and database queues drained.
• Bounded Redis failures: Deployed a Redis-client change to detect stalled established connections, bound failed operations, and reconnect more promptly.
• Corrected the decryption fallback: Changed the migration flag’s fallback to behavior compatible with both current and legacy ciphertext formats.
• Validated the recovery fix: Tested stalled-client behavior and an ElastiCache failover in staging before deploying the connection changes to production.

Near-term Improvements

The following work remains open:

• Extend bounded connection recovery to other Redis clients and verify their behavior during failover (in progress).
• Remove feature-flag and decryption-service network waits from authentication database transactions (in progress).
• Complete the legacy decryption migration and retire its obsolete flag and fallback paths.
• Audit feature-flag defaults against stable production behavior and retire flags whose migrations are complete.
• Evaluate and pilot streaming flag updates into application memory, reducing the need to contact Redis while serving requests.

Preventive Measures

Our follow-up work includes:

• Improving visibility into flag-evaluation timeouts, fallback use, and connection recovery.
• Adding checks for external network waits inside database transactions.
• Requiring affected service components to be selected when opening a status-page incident, so component-specific subscribers receive notificati