{"id":"INC-github-dnslv7xdl4fj","kind":"incident","aliases":[],"related":[],"title":"GitHub: Elevated 404s for GitHub App s2s tokens in git/API","description":"Between 0655 and 0801 UTC on 10/27, customers experienced increased failure rates for API and git requests authenticated with server-to-server tokens for GitHub Apps (excluding Actions tokens). During this window, authenticated git operations mistakenly returned 404 for 1.24% (2.6x) of requests on average and 2.49% (5.3x) at peak.\n\nThese failures were caused by a scheduled maintenance event where GitHub migrated 7 tables between two MySQL clusters in our IAD and Azure Central US datacenters. One of the affected tables stores installation records for GitHub App server-to-server tokens (excluding Actions). We've successfully completed several of these migrations in recent months without notable customer impact. However, in this case, the migration coincided with a scheduled cleanup job on the installation records table, which deleted approximately 59 million records from 0444-0617 UTC. As a result, the gh-ost tooling used in the migration experienced inter-cluster replication delays (up to 1 hour at peak).\n\nAt 0655 UTC, we begin transitioning reads of the target tables to the new MySQL cluster, with 100% of reads cutover at 0721 UTC. Customer impact began, but was not detected, throughout the read cutover and persisted until completion of the write cutover at 0801 UTC. We are updating our processes and automation to prevent a repeat of this incident, including disabling overlapping cleanup jobs prior to these migrations and blocking the cutover of reads when replication delays are high.","summary":"Between 0655 and 0801 UTC on 10/27, a database migration at GitHub caused authenticated API and git requests using GitHub App server-to-server tokens (excluding Actions tokens) to return false 404 errors. The incident was an availability problem, and GitHub reports it as resolved, with no attacker access involved.","severity":"low","cvss":null,"epss":null,"exploited":false,"exploited_at":null,"products":[{"name":"GitHub","fixed":[],"affected":"","ecosystem":"vendor"}],"ecosystems":["vendor"],"tags":["ci","git","github","github-apps","incident","security","vendor"],"not_affected_if":["actions-tokens-only","no-github-app-tokens"],"how_to_check":null,"fix":null,"status":"resolved","published_at":"2026-10-27T08:00:00+00:00","first_seen_at":"2026-10-27T20:24:38+00:00","triaged_at":"2026-10-27T20:24:56+00:00","updated_at":"2026-10-27T20:24:56+00:00","category":"vendor","confidence":0.8,"triage":{"at":"2026-10-27T20:24:56Z","model":"claude-haiku-5-5","output":{"fix":"unknown","reach":"internet","title":"GitHub: outage returned 404 for GitHub App server-to-server token requests","impact":"low","summary":"Between 0655 and 0801 UTC on 10/27, a database migration at GitHub caused authenticated API and git requests using GitHub App server-to-server tokens (excluding Actions tokens) to return false 404 errors. The incident was an availability problem, and GitHub reports it as resolved, with no attacker access involved.","category":"vendor","nicknames":[],"who_cares":["github","ci","github-apps"],"confidence":0.8,"needs_auth":"no","worst_case":"Authenticated git and API calls fail with false 404 errors for about 1 hour.","analyst_note":"","how_to_check":"unknown","needs_analyst":false,"not_affected_if":["actions-tokens-only","no-github-app-tokens"],"who_is_affected":"Teams whose automation or CI authenticates to GitHub with GitHub App server-to-server tokens (excluding Actions tokens) saw failed git and API calls during the window. The sources describe a resolved outage with no data exposure, so no action is required beyond reviewing failed jobs from that period.","user_interaction":"no","needs_special_setup":"no","severity_if_unknown":"unknown"},"cost_usd":0.000596,"input_tokens":3716,"output_tokens":448,"severity_before":"low"},"analyst_status":null,"analyst_note":null,"search":"'-0617':213C '0444':212C '0655':26B,77C,240C '0721':261C '0801':28B,79C,283C '1':235C '1.24':116C '10/27':31B,82C '100':256C '2.49':124C '2.6':117C '404':57B,114C '404s':3A '5.3':125C '59':208C '7':141C 'access':74B 'action':52B,104C,172C 'affect':158C 'api':39B,89C 'app':6A,21B,45B,102C,165C 'approxim':207C 'attack':73B 'authent':38B,93C,109C 'autom':291C 'avail':63B 'averag':122C 'azur':151C 'began':265C 'begin':243C 'block':309C 'case':191C 'caus':37B,132C 'central':152C 'ci':16B 'cleanup':198C,302C 'cluster':146C,230C,254C 'coincid':194C 'complet':176C,277C 'custom':83C,186C,263C 'cutov':259C,273C,281C,311C 'databas':33B 'datacent':154C 'delay':232C,316C 'delet':206C 'detect':269C 'disabl':300C 'dnslv7xdl4fj':14A 'elev':2A 'error':58B 'event':137C 'exclud':51B,103C,171C 'experienc':84C,227C 'failur':86C,130C 'fals':56B 'gh':220C 'gh-ost':219C 'git':17B,41B,91C,110C 'git/api':10A 'github':1A,5A,13A,15A,18B,20B,36B,44B,66B,101C,139C,164C 'github-app':19B 'high':318C 'hour':236C 'howev':188C 'iad':149C 'impact':187C,264C 'inc':12A 'inc-github-dnslv7xdl4fj':11A 'incid':22B,60B,298C 'includ':299C 'increas':85C 'instal':161C,202C 'inter':229C 'inter-clust':228C 'involv':75B 'job':199C,303C 'mainten':136C 'migrat':34B,140C,180C,193C,226C,307C 'million':209C 'mistaken':112C 'month':183C 'mysql':145C,253C 'new':252C 'notabl':185C 'one':155C 'oper':111C 'ost':221C 'overlap':301C 'peak':128C,238C 'persist':275C 'prevent':293C 'prior':304C 'problem':64B 'process':289C 'rate':87C 'read':245C,258C,272C,313C 'recent':182C 'record':162C,203C,210C 'repeat':295C 'replic':231C,315C 'report':67B 'request':42B,92C,120C 'resolv':70B 'result':217C 'return':55B,113C 's2s':7A 'schedul':135C,197C 'secur':23B 'server':47B,49B,96C,98C,167C,169C 'server-to-serv':46B,95C,166C 'sever':177C 'store':160C 'success':175C 'tabl':142C,159C,204C,249C 'target':248C 'throughout':270C 'token':8A,50B,53B,99C,105C,170C 'tool':222C 'transit':244C 'two':144C 'updat':287C 'us':153C 'use':43B,223C 'utc':29B,80C,214C,241C,262C,284C 've':174C 'vendor':24B 'window':108C 'without':184C 'write':280C 'x':118C,126C","nicknames":[],"mitigations":[],"who_affected":"Teams whose automation or CI authenticates to GitHub with GitHub App server-to-server tokens (excluding Actions tokens) saw failed git and API calls during the window. The sources describe a resolved outage with no data exposure, so no action is required beyond reviewing failed jobs from that period.","tier":"C","verified":{},"analyst_revisit_at":null,"first_public_at":"2026-10-27T08:00:00+00:00","kev":null,"cvss_vector":null,"cvss_by":null,"poc_count":0,"sources":[{"source":"statuspage","external_id":"github:dnslv7xdl4fj","url":"https://stspg.io/4z5lmy7gqjk1","severity":"low","published_at":"2026-10-27T08:00:00+00:00","modified_at":"2026-10-27T20:21:53+00:00","first_seen_at":"2026-10-27T20:24:38+00:00","fetched_at":"2026-10-27T22:14:26+00:00","note":null}],"documents":[],"notice":"Free to read at a human pace. An API with keys and an MCP server are in preparation; early access: hello@infrastack.ai"}