---
id: "INC-pypi-r47c43jw19zq"
kind: "incident"
aliases: []
title: "PyPI: Python Infrastructure Outage"
severity: "critical"
cvss: null
epss: null
exploited: false
status: "resolved"
category: "vendor"
products: [{"name": "PyPI", "fixed": [], "affected": "", "ecosystem": "vendor"}]
ecosystems: ["vendor"]
tags: ["incident", "ops", "pypi", "python", "registry", "vendor"]
not_affected_if: []
published_at: "2026-02-26T20:04:21+00:00"
first_seen_at: "2026-10-23T19:12:22+00:00"
updated_at: "2026-10-25T05:37:22+00:00"
sources: [{"source": "statuspage", "url": "https://stspg.io/7slh65j2y14b", "first_seen_at": "2026-10-23T19:12:22+00:00"}]
---

# PyPI: Python Infrastructure Outage

## Affected

- vendor:PyPI affected ? fixed ?

## Timeline

- 2026-10-23T19:12:22+00:00 statuspage https://stspg.io/7slh65j2y14b

## From the sources

The incident has been resolved.

## Incident Report: Cluster-wide service outage

**Duration**: ~36 minutes \(20:04–20:40 UTC\)

**Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. 

What was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI.

**Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters.

The scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures.

**Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history.

Action items:

* Separate kubeconfig files for production vs local, rather than relying on context switching
* Research adding admission control or policies to prevent bulk scale-to-zero operations
* Document the infrastructure dependency chain and recovery runbook for future incidents

Jacob Coffee, PSF Infrastructure  Team

---

_Free to read at a human pace. An API with keys and an MCP server are in preparation; early access: hello@infrastack.ai_