{"id":"INC-pypi-r47c43jw19zq","kind":"incident","aliases":[],"related":[],"title":"PyPI: Python Infrastructure Outage","description":"The incident has been resolved.\n\n## Incident Report: Cluster-wide service outage\n\n**Duration**: ~36 minutes \\(20:04–20:40 UTC\\)\n\n**Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. \n\nWhat was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI.\n\n**Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters.\n\nThe scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures.\n\n**Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history.\n\nAction items:\n\n* Separate kubeconfig files for production vs local, rather than relying on context switching\n* Research adding admission control or policies to prevent bulk scale-to-zero operations\n* Document the infrastructure dependency chain and recovery runbook for future incidents\n\nJacob Coffee, PSF Infrastructure  Team","summary":null,"severity":"critical","cvss":null,"epss":null,"exploited":false,"exploited_at":null,"products":[{"name":"PyPI","fixed":[],"affected":"","ecosystem":"vendor"}],"ecosystems":["vendor"],"tags":["incident","ops","pypi","python","registry","vendor"],"not_affected_if":[],"how_to_check":null,"fix":null,"status":"resolved","published_at":"2026-02-26T20:04:21+00:00","first_seen_at":"2026-10-23T19:12:22+00:00","triaged_at":"2026-10-25T05:37:22+00:00","updated_at":"2026-10-25T05:37:22+00:00","category":"vendor","confidence":null,"triage":null,"analyst_status":null,"analyst_note":null,"search":"'04':32C '20':31C,33C '36':29C '40':34C 'action':149C 'add':165C 'admiss':166C 'among':67C 'bring':134C 'bugs.python.org':51C,52C 'bulk':172C 'cascad':121C 'caus':74C 'chain':182C 'cluster':24C,62C,93C,103C 'cluster-wid':23C 'coffe':190C 'command':98C 'context':86C,162C 'control':167C 'count':142C 'creat':120C 'depend':138C,181C 'deploy':112C 'develop':77C 'document':178C 'durat':28C 'durbin':132C 'ee':131C 'environ':108C 'event':147C 'failur':122C 'file':153C 'futur':187C 'help':129C 'histori':148C 'host':40C 'impact':36C 'inc':6A 'inc-pypi-r47c43jw19zq':5A 'incid':10B,17C,21C,188C 'includ':44C 'incorrect':85C 'infrastructur':3A,136C,180C,192C 'instead':104C 'item':150C 'iter':109C 'jacob':189C 'kubeconfig':152C 'kubernet':79C,146C 'local':76C,81C,107C,157C 'manag':64C 'minut':30C 'one':89C 'op':11B 'oper':177C 'order':139C 'origin':140C 'outag':4A,27C 'polici':169C 'prevent':171C 'product':92C,102C,155C 'psf':39C,191C 'psf-host':38C 'pypi':1A,7A,9A,12B,49C,72C 'pypi.org':65C,66C 'python':2A,13B 'python.org':45C,46C 'r47c43jw19zq':8A 'ran':99C 'rather':158C 'recov':144C 'recoveri':123C,184C 'registri':14B 'relat':54C,70C 'reli':160C 'replica':118C,141C 'report':22C 'research':164C 'resolv':20C 'restor':126C 'root':73C 'runbook':185C 'scale':96C,174C 'scale-down':95C 'scale-to-zero':173C 'separ':151C 'servic':26C,41C,55C,69C,124C 'set':114C 'stat':50C 'switch':87C,163C 'team':193C 'unaffect':58C 'unavail':43C 'us.pycon.org':47C,48C 'utc':35C 'vendor':15B 'vs':156C 'wide':25C 'workload':80C 'zero':117C,176C","nicknames":[],"mitigations":[],"who_affected":null,"tier":null,"verified":{},"analyst_revisit_at":null,"first_public_at":"2026-02-26T20:04:21+00:00","kev":null,"cvss_vector":null,"cvss_by":null,"poc_count":0,"sources":[{"source":"statuspage","external_id":"pypi:r47c43jw19zq","url":"https://stspg.io/7slh65j2y14b","severity":"critical","published_at":"2026-02-26T20:04:21+00:00","modified_at":"2026-02-26T23:27:22+00:00","first_seen_at":"2026-10-23T19:12:22+00:00","fetched_at":"2026-10-25T18:56:18+00:00","note":null}],"documents":[],"notice":"Free to read at a human pace. An API with keys and an MCP server are in preparation; early access: hello@infrastack.ai"}