---
id: "INC-trigger-939294"
page: "https://infrastack.ai/live/INC-trigger-939294"
kind: "incident"
aliases: []
title: "Trigger: Run Lists and Logs Are Degraded"
severity: "unknown"
cvss: null
epss: null
exploited: false
status: "open"
category: "vendor"
products: [{"name": "Trigger", "fixed": [], "affected": "", "ecosystem": "vendor"}]
ecosystems: ["vendor"]
tags: ["incident", "ops", "trigger", "vendor"]
not_affected_if: []
published_at: "2026-06-30T09:40:00+00:00"
first_seen_at: "2026-11-08T16:58:31+00:00"
updated_at: "2026-11-09T02:36:08+00:00"
sources: [{"source": "statuspage", "url": "https://status.trigger.dev/incident/939294", "first_seen_at": "2026-11-08T16:58:31+00:00"}]
exploit_sources: []
---

# Trigger: Run Lists and Logs Are Degraded

Link for people: https://infrastack.ai/live/INC-trigger-939294 (this .md is for agents)

## Affected

- vendor:Trigger affected range not listed fixed no fixed version listed

## Timeline

- 2026-11-08T16:58:31+00:00 statuspage https://status.trigger.dev/incident/939294

## From the sources

First, the important part: your runs are safe. Runs execute and are stored in Postgres as normal. This issue is isolated to ClickHouse, which we use as a read-optimised view store for querying runs. Triggering, execution, and storage have been unaffected throughout. This is a visibility issue, not a data loss one.

The fix is downgrading ClickHouse to 25.12. That rollback runs at the ClickHouse Cloud infrastructure level, so we're coordinating with them to apply it. It's agreed and queued.

## What you're seeing

While the view was degraded, the runs list, filtering, bulk actions, the query API, and all error monitoring and alerting were slow or unavailable, and dashboard charts, traces, logs, and span detail were blank or incomplete for affected runs. Run execution, billing, and your Postgres-stored run data were unaffected throughout.

- Dashboard and API queries against run data are slow or failing. There was a window this morning where these reads were down entirely.
- A small percentage of runs created since June 17 aren't yet in the correct state in the view. The runs ran and are stored. The view of them is catching up.
- Run logs and spans have recovered. Run views are still catching up.

## What's causing it

ClickHouse moved from 25.12 to 26.2 on June 17. The new version added a cap on JSON column complexity (`input_format_binary_max_type_complexity`, defaulting to 1000), measured across the combined outputs in each batched insert. Once a batch went over, those inserts stopped landing in the view. Then this morning, June 30, memory and CPU on the instance climbed sharply as the backlog of unmerged data caught up with it, driving reads down until it ran out of memory and caused today's outage. The 25.12 rollback resolves the root cause.

## What we've done so far

- Cleared the data that couldn't merge so healthy merges can continue. Run logs and spans recovered quickly; run views are slower and still catching up.
- Scaled the cluster horizontally to add headroom.
- Lined up the 25.12 rollback with ClickHouse Cloud.

---

_Free to read, with fair use of about 60 pages a minute. For higher limits or API access: sales@infrastack.ai_