{"id":"INC-apify-4k6n5swkwx5h","kind":"incident","aliases":[],"related":[],"title":"Apify: Major systems outage","description":"Saturday outage post-mortem  \nDate of incident: 2025-02-22\n\nImpact: \n\n* Complete downtime of the Apify platform from 22.00 UTC to 23.00 UTC.\n* No Actors could be started. \n* Those already running were paused and continued running after the incident was resolved. \n* There was no data loss, but the workloads were disrupted.\n\nWhat happened:\n\n* Our user base is growing fast, and the number of new signups is increasing exponentially. The exponentially increased load revealed suboptimalities that had not manifested before.\n* One misconfigured query then caused a peak in memory due to a load of a suboptimal index, which caused the cluster to crash.\n\nWhat we did:\n\n* The cluster size was increased to ensure enough memory until the issue was fixed. \n* Then, the cluster and all the dependent systems were restarted.\n\nNext steps:\n\n* We are already re-architecting our primary database clusters to optimize systems for future growth.\n* As part of this, we aim to decrease the size of all our clusters to ensure performant restarts in case of a problem.\n* We are currently optimizing indexing and querying among heavy-load components.\n* In addition, we identified missing metrics that would indicate similar problems ahead of time.\n\nWe sincerely apologize for the disruption and appreciate your patience. If you have any questions, please reach out to [support@apify.com](mailto:support@apify.com).\n\nSincerely,  \nApify engineering team","summary":null,"severity":"critical","cvss":null,"epss":null,"exploited":false,"exploited_at":null,"products":[{"name":"Apify","fixed":[],"affected":"","ecosystem":"vendor"}],"ecosystems":["vendor"],"tags":["apify","incident","ops","vendor"],"not_affected_if":[],"how_to_check":null,"fix":null,"status":"resolved","published_at":"2025-02-22T22:12:18+00:00","first_seen_at":"2026-11-12T15:06:32+00:00","triaged_at":"2026-11-12T15:07:42+00:00","updated_at":"2026-11-12T15:07:42+00:00","category":"vendor","confidence":null,"triage":null,"analyst_status":null,"analyst_note":null,"search":"'-02':23C '-22':24C '2025':22C '22.00':33C '23.00':36C '4k6n5swkwx5h':8A 'actor':39C 'addit':198C 'ahead':208C 'aim':167C 'alreadi':44C,148C 'among':192C 'apifi':1A,10B,30C,234C 'apify':7A,9A 'apolog':213C 'appreci':218C 'architect':151C 'base':70C 'case':181C 'caus':98C,112C 'cluster':114C,121C,136C,155C,175C 'complet':26C 'compon':196C 'continu':49C 'could':40C 'crash':116C 'current':187C 'data':59C 'databas':154C 'date':19C 'decreas':169C 'depend':140C 'disrupt':65C,216C 'downtim':27C 'due':103C 'engin':235C 'enough':127C 'ensur':126C,177C 'exponenti':82C,84C 'fast':73C 'fix':133C 'futur':160C 'grow':72C 'growth':161C 'happen':67C 'heavi':194C 'heavy-load':193C 'identifi':200C 'impact':25C 'inc':6A 'inc-apify-4k6n5swkwx5h':5A 'incid':11B,21C,53C 'increas':81C,85C,124C 'index':110C,189C 'indic':205C 'issu':131C 'load':86C,106C,195C 'loss':60C 'mailto':231C 'major':2A 'manifest':92C 'memori':102C,128C 'metric':202C 'misconfigur':95C 'miss':201C 'mortem':18C 'new':78C 'next':144C 'number':76C 'one':94C 'op':12B 'optim':157C,188C 'outag':4A,15C 'part':163C 'patienc':220C 'paus':47C 'peak':100C 'perform':178C 'platform':31C 'pleas':226C 'post':17C 'post-mortem':16C 'primari':153C 'problem':184C,207C 'queri':96C,191C 'question':225C 're':150C 're-architect':149C 'reach':227C 'resolv':55C 'restart':143C,179C 'reveal':87C 'run':45C,50C 'saturday':14C 'signup':79C 'similar':206C 'sincer':212C,233C 'size':122C,171C 'start':42C 'step':145C 'suboptim':88C,109C 'support@apify.com':230C,232C 'system':3A,141C,158C 'team':236C 'time':210C 'user':69C 'utc':34C,37C 'vendor':13B 'workload':63C 'would':204C","nicknames":[],"mitigations":[],"who_affected":null,"tier":null,"verified":{},"analyst_revisit_at":null,"fix_status":null,"audience":null,"plain_action":null,"pkeys":["apify"],"ptokens":["apify"],"labels":["apify","critical","incident","ops","vendor"],"campaign_type":null,"search_a":"'4k6n5swkwx5h':8A 'apifi':1A 'apify':7A,9A 'inc':6A 'inc-apify-4k6n5swkwx5h':5A 'major':2A 'outag':4A 'system':3A","search_b":"'apifi':10B 'incid':11B 'op':12B 'vendor':13B","search_c":"'-02':23C '-22':24C '2025':22C '22.00':33C '23.00':36C 'actor':39C 'addit':198C 'ahead':208C 'aim':167C 'alreadi':44C,148C 'among':192C 'apifi':30C,234C 'apolog':213C 'appreci':218C 'architect':151C 'base':70C 'case':181C 'caus':98C,112C 'cluster':114C,121C,136C,155C,175C 'complet':26C 'compon':196C 'continu':49C 'could':40C 'crash':116C 'current':187C 'data':59C 'databas':154C 'date':19C 'decreas':169C 'depend':140C 'disrupt':65C,216C 'downtim':27C 'due':103C 'engin':235C 'enough':127C 'ensur':126C,177C 'exponenti':82C,84C 'fast':73C 'fix':133C 'futur':160C 'grow':72C 'growth':161C 'happen':67C 'heavi':194C 'heavy-load':193C 'identifi':200C 'impact':25C 'incid':21C,53C 'increas':81C,85C,124C 'index':110C,189C 'indic':205C 'issu':131C 'load':86C,106C,195C 'loss':60C 'mailto':231C 'manifest':92C 'memori':102C,128C 'metric':202C 'misconfigur':95C 'miss':201C 'mortem':18C 'new':78C 'next':144C 'number':76C 'one':94C 'optim':157C,188C 'outag':15C 'part':163C 'patienc':220C 'paus':47C 'peak':100C 'perform':178C 'platform':31C 'pleas':226C 'post':17C 'post-mortem':16C 'primari':153C 'problem':184C,207C 'queri':96C,191C 'question':225C 're':150C 're-architect':149C 'reach':227C 'resolv':55C 'restart':143C,179C 'reveal':87C 'run':45C,50C 'saturday':14C 'signup':79C 'similar':206C 'sincer':212C,233C 'size':122C,171C 'start':42C 'step':145C 'suboptim':88C,109C 'support@apify.com':230C,232C 'system':141C,158C 'team':236C 'time':210C 'user':69C 'utc':34C,37C 'workload':63C 'would':204C","cause_entry":null,"cause_path":null,"disclosures_checked_at":null,"first_public_at":"2025-02-22T22:12:18+00:00","kev":null,"cvss_vector":null,"cvss_by":null,"poc_count":0,"sources":[{"source":"statuspage","external_id":"apify:4k6n5swkwx5h","url":"https://stspg.io/mlqbfmm8klx1","severity":"critical","published_at":"2025-02-22T22:12:18+00:00","modified_at":"2025-02-26T14:21:30+00:00","first_seen_at":"2026-11-12T15:06:32+00:00","fetched_at":"2026-11-12T15:57:58+00:00","note":null}],"documents":[],"disclosures":[{"company":"Apify","domain":"apify.com","url":"https://stspg.io/7sp049kjvxqb","kind":"statuspage","published_at":"2026-11-11T19:46:21+00:00","quote":"On 9 and 10 November 2026, an unauthorized third party gained access to parts of Apify’s internal systems.","confidence":0.9700000286102295,"source":"statuspage","created_at":"2026-11-12T15:05:57+00:00","entry":"stolen-credential"},{"company":"Apify","domain":"stspg.io","url":"https://stspg.io/7sp049kjvxqb","kind":"statuspage","published_at":"2026-11-11T19:46:21+00:00","quote":"On 9 and 10 November, an unauthorized third party used a compromised credential to obtain a number of account credentials and accessed related Apify accounts.","confidence":0.9700000286102295,"source":"statuspage","created_at":"2026-11-12T15:15:49+00:00","entry":"stolen-credential"}],"exploit_sources":[],"notice":"Free to read, with fair use of about 60 pages a minute. For higher limits or API access: sales@infrastack.ai"}