Trigger: us-east-1 region dequeue degradation

Our advice remains the same: switch your default region to us-east-1 if you can. More details below.

us-east-1 Operating normally with global queue times better than normal thresholds. If you have excessive runs queued in us-east-1:

1. Check your concurrency limits, increase as needed including at the account level from the Concurrency page (let us know via the in-app Help and Feedback and we will credit your account)
2. If you were on eu-central-1 they could be blocking your us-east-1 runs. Go to the Runs page and filter for "Queued" status and "eu-central-1" region. Then use the Bulk action on the right to cancel those runs.
3. When eu-central-1 is back you can replay these runs there.

eu-central-1 The cluster is being spun back up with more resources and the same settings we used to fix us-east-1. Currently no runs are executing here.

The same thundering herd issue that happened in us-east-1 caused this issue as everyone moved their traffic over. This cluster was smaller so it took less to cause this problem.

--

After this incident is over we will do a full post mortem with what happened along with a long list of things we need to do better. I've been noting them down and we've recorded this entire process.

We will also be offering credits to all impacted customers. I know this doesn't cover the cost to your businesses.

This is far from the bar we set for reliability. Sorry everyone, Matt