Trigger: Runs list/logs degraded

Postmortem: Missing task logs and traces

Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand.

What customers experienced

Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected.

Root cause

A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert.