{"id":"INC-pypi-bt0tzq7x2fk2","kind":"incident","aliases":[],"related":[],"title":"PyPI: Newly uploaded files are returning 403 from files.pythonhosted.org","description":"## Summary\n\nThe cloud infrastructure that runs PyPI is managed via an automation tool which allows for configuration changes to be version controlled, reviewed, and deployed automatically. This tool includes multiple backends for state sharing and locking to facilitate collaboration between multiple contributors. While migrating the remote state backend and locking features to a more robust hosted platform an error was made when setting the credentials that grant our CDN access to the object storage where uploaded files to PyPI are hosted.\n\nThe result was that newly uploaded files as well as files that had expired from our CDNs cache were unavailable. The outage began 2022-06-022 20:28 UTC and was initially mitigated 2022-06-02 22:38 UTC by rolling back the configuration to a previous version. The configuration issue in the hosted platform was confirmed and corrected 2022-06-02 12:00 UTC and a corrected new version of the CDN configuration was deployed. Future deploys will include this value and will not regress in the same manner.\n\n## Details\n\nAs the Infrastructure team at the PSF onboards a new hire and the clunkiness of collaborating on our terraform configuration led to only two people having the ability to deploy changes, a decision was made to migrate all state, sensitive variables, and locking to a hosted backend for terraform. This will allow all those with appropriate access \\(PyPI Admins and PSF Infrastructure Staff\\) access without having to pass around plaintext secrets necessary to plan and apply changes to the infrastructure.\n\nDuring the migration from S3 state storage and DynamoDB locking to Hashicorp Terraform Cloud, the Infrastructure staff lead made an error in copying the value for the GCS secret access key that grants Fastly, our CDN, access to read the Cloud Storage bucket where PyPI uploads are stored.\n\nBecause the value was marked as sensitive in Terraform Cloud via a setting, it wasn’t obvious to the staff member when planning the changes what would change, and it was assumed to be an artifact of that setting. As a confirmation and for assurances, the staff member configured the changes to Fastly configuration to _not_ auto activate the new configuration.\n\nAfter applying the changes which would not be automatically activated in Fastly it is unclear why the incorrect change was not identified when diffing service configurations, but ultimately it was missed and the services were activated.\n\nImpact lasted 130 minutes until enough signal from users was raised to alert PyPI admins and affected all new uploaded files as well as any files that had expired from our CDN caches.\n\n## Future Mitigation\n\nThis outage highlighted two key areas where improvements can be made to ensure timely response\n\n1. **More thorough automated monitoring of file access via our CDN.** Due to the caching nature of our CDN, it was not detected that this error was occurring as the file checked never expired from cache. We will implement a monitor that checks for a file that is explicitly not cached to ensure that the entire system from client to object-store is working appropriately\n2. **Improvements to secrets handling in our infrastructure automation.** We carried some legacy practices for storing secrets/configuration in our systems that led to pertinent configuration value being obscured in a complex object rather than an individual key. We’ll refactor this to make detecting changes to values more obvious to contributors to the project.","summary":"A misconfigured CDN credential caused files.pythonhosted.org to return 403 for newly uploaded PyPI files and expired cached files. The incident was an availability outage with no sign of data exposure or attacker access in the source.","severity":"medium","cvss":null,"epss":null,"exploited":false,"exploited_at":null,"products":[{"name":"PyPI","fixed":[],"affected":"","ecosystem":"vendor"}],"ecosystems":["vendor"],"tags":["incident","pypi","python","registry","security","vendor"],"not_affected_if":["no-pypi-installs-during-window"],"how_to_check":null,"fix":null,"status":"resolved","published_at":"2022-06-02T22:20:20+00:00","first_seen_at":"2026-10-23T19:12:22+00:00","triaged_at":"2026-10-25T05:41:51+00:00","updated_at":"2026-10-25T05:41:51+00:00","category":"vendor","confidence":0.6,"triage":{"at":"2026-10-25T05:41:51Z","model":"claude-haiku-5-5","output":{"fix":"unknown","reach":"internet","title":"PyPI: newly uploaded package files returned 403 for about 130 minutes","impact":"low","summary":"A misconfigured CDN credential caused files.pythonhosted.org to return 403 for newly uploaded PyPI files and expired cached files. The incident was an availability outage with no sign of data exposure or attacker access in the source.","category":"vendor","nicknames":[],"who_cares":["python","pypi"],"confidence":0.6,"needs_auth":"no","worst_case":"Package downloads fail with 403 errors; no code execution or data exposure reported.","analyst_note":"","how_to_check":"unknown","needs_analyst":false,"not_affected_if":["no-pypi-installs-during-window"],"who_is_affected":"Teams that install packages from PyPI during the outage window (2022-06-02, roughly 20:28 to 22:38 UTC) could have seen failed installs of newly uploaded or uncached files. The source describes the outage as resolved and does not describe any action for package consumers to take.","user_interaction":"no","needs_special_setup":"no","severity_if_unknown":"unknown"},"cost_usd":0.00063,"input_tokens":4300,"output_tokens":399,"severity_before":"medium"},"analyst_status":null,"analyst_note":null,"search":"'-02':173C,199C '-022':163C '-06':162C,172C,198C '00':201C '1':504C '12':200C '130':456C '2':570C '20':164C '2022':161C,171C,197C '22':174C '28':165C '38':175C '403':7A,29B 'abil':256C 'access':53B,126C,285C,292C,338C,345C,511C 'activ':414C,427C,453C 'admin':287C,468C 'affect':470C 'alert':466C 'allow':71C,280C 'appli':304C,419C 'appropri':284C,569C 'area':494C 'around':297C 'artifact':392C 'assum':388C 'assur':401C 'attack':52B 'auto':413C 'autom':68C,507C,578C 'automat':82C,426C 'avail':43B 'back':179C 'backend':87C,104C,275C 'began':160C 'bt0tzq7x2fk2':13A 'bucket':351C 'cach':37B,155C,486C,518C,539C,554C 'carri':580C 'caus':25B 'cdn':23B,125C,210C,344C,485C,514C,522C 'cdns':154C 'chang':74C,259C,305C,381C,384C,407C,421C,436C,614C 'check':535C,546C 'client':562C 'cloud':59C,322C,349C,366C 'clunki':242C 'collabor':95C,244C 'complex':600C 'configur':73C,181C,187C,211C,248C,405C,410C,417C,443C,594C 'confirm':194C,398C 'contributor':98C,620C 'control':78C 'copi':331C 'correct':196C,205C 'credenti':24B,121C 'data':49B 'decis':261C 'deploy':81C,213C,215C,258C 'detail':228C 'detect':526C,613C 'dif':441C 'due':515C 'dynamodb':317C 'enough':459C 'ensur':501C,556C 'entir':559C 'error':115C,329C,529C 'expir':36B,151C,482C,537C 'explicit':552C 'exposur':50B 'facilit':94C 'fast':342C,409C,429C 'featur':107C 'file':4A,34B,38B,133C,144C,148C,474C,479C,510C,534C,549C 'files.pythonhosted.org':9A,26B 'futur':214C,487C 'gcs':336C 'grant':123C,341C 'handl':574C 'hashicorp':320C 'highlight':491C 'hire':239C 'host':112C,137C,191C,274C 'identifi':439C 'impact':454C 'implement':542C 'improv':496C,571C 'inc':11A 'inc-pypi-bt0tzq7x2fk2':10A 'incid':15B,40B 'includ':85C,217C 'incorrect':435C 'individu':605C 'infrastructur':60C,231C,290C,308C,324C,577C 'initi':169C 'issu':188C 'key':339C,493C,606C 'last':455C 'lead':326C 'led':249C,591C 'legaci':582C 'll':608C 'lock':92C,106C,271C,318C 'made':117C,263C,327C,499C 'make':612C 'manag':65C 'manner':227C 'mark':361C 'member':377C,404C 'migrat':100C,265C,311C 'minut':457C 'misconfigur':22B 'miss':448C 'mitig':170C,488C 'monitor':508C,544C 'multipl':86C,97C 'natur':519C 'necessari':300C 'never':536C 'new':206C,238C,416C,472C 'newli':2A,31B,142C 'object':129C,565C,601C 'object-stor':564C 'obscur':597C 'obvious':373C,618C 'occur':531C 'onboard':236C 'outag':44B,159C,490C 'pass':296C 'peopl':253C 'pertin':593C 'plaintext':298C 'plan':302C,379C 'platform':113C,192C 'practic':583C 'previous':184C 'project':623C 'psf':235C,289C 'pypi':1A,12A,14A,16B,33B,63C,135C,286C,353C,467C 'python':17B 'rais':464C 'rather':602C 'read':347C 'refactor':609C 'registri':18B 'regress':223C 'remot':102C 'respons':503C 'result':139C 'return':6A,28B 'review':79C 'robust':111C 'roll':178C 'run':62C 's3':313C 'secret':299C,337C,573C 'secrets/configuration':586C 'secur':19B 'sensit':268C,363C 'servic':442C,451C 'set':119C,369C,395C 'share':90C 'sign':47B 'signal':460C 'sourc':56B 'staff':291C,325C,376C,403C 'state':89C,103C,267C,314C 'storag':130C,315C,350C 'store':356C,566C,585C 'summari':57C 'system':560C,589C 'team':232C 'terraform':247C,277C,321C,365C 'thorough':506C 'time':502C 'tool':69C,84C 'two':252C,492C 'ultim':445C 'unavail':157C 'unclear':432C 'upload':3A,32B,132C,143C,354C,473C 'user':462C 'utc':166C,176C,202C 'valu':219C,333C,359C,595C,616C 'variabl':269C 'vendor':20B 'version':77C,185C,207C 'via':66C,367C,512C 'wasn':371C 'well':146C,476C 'without':293C 'work':568C 'would':383C,423C","nicknames":[],"mitigations":[],"who_affected":"Teams that install packages from PyPI during the outage window (2022-06-02, roughly 20:28 to 22:38 UTC) could have seen failed installs of newly uploaded or uncached files. The source describes the outage as resolved and does not describe any action for package consumers to take.","tier":"C","verified":{},"analyst_revisit_at":null,"first_public_at":"2022-06-02T22:20:20+00:00","kev":null,"cvss_vector":null,"cvss_by":null,"poc_count":0,"sources":[{"source":"statuspage","external_id":"pypi:bt0tzq7x2fk2","url":"https://stspg.io/8fzwjz86r0l1","severity":"medium","published_at":"2022-06-02T22:20:20+00:00","modified_at":"2022-06-03T12:33:37+00:00","first_seen_at":"2026-10-23T19:12:22+00:00","fetched_at":"2026-10-25T23:42:33+00:00","note":null}],"documents":[],"notice":"Free to read at a human pace. An API with keys and an MCP server are in preparation; early access: hello@infrastack.ai"}