# Queue storm / 504s

**Symptom:** a queue topic (usually `run-chart-review` or `run-triggered-chart-review`) returns 504s in a loop. Retries pile up, throughput drops, and LLM/Pinecone 429s rise alongside.

## Why it happens

- A chart review runs 600–800s, right at the 800s `maxDuration`. Any added latency tips it into a 504.
- High `maxConcurrency` saturates the shared LLM rate limit. Everything slows past 800s together, so throughput *falls* as concurrency rises.
- **Messages are pinned to the deployment that enqueued them.** A fix you deploy only applies to messages enqueued after it. The storm keeps hitting the old deployment's code and config.

Unpinning on send was tried and rejected: it lets release N+1 run release N's half-finished jobs. Pinning stays; deleting the stale deployment is the accepted stop.

## Confirm

1. Datadog APM: filter the failing jobbable by status. Note the **deployment ID / host** serving the 504s.
2. Compare it with the current production deployment in Vercel. If it differs, deploying a fix won't help the queued messages.
3. Don't look for the storm in `job_executions`: dedup keeps one row per logical job, and a 504-killed job stays `running`. Count 504s in Datadog or Vercel.

## Stop it

1. **Fix forward for new messages:** lower the topic's `maxConcurrency` in `apps/web/vercel.json` and ship it. Keep `retryAfterSeconds` ≥ `maxDuration` (800), or a still-running job is redelivered as a concurrent duplicate.
2. **Stop the old messages:** if 504s continue from a non-current deployment, delete that deployment in Vercel. Its queued messages are lost; that is the trade.
3. Post in the on-call channel what you deleted (deployment ID and commit).

## Recover

1. Superadmin → Audit runs: filter to runs from the storm window that aren't complete, then "Re-run all questions". Answers are saved per question, so a re-run picks up where the last one died instead of timing out at the same point.
2. The cleanup cron fails runs left `running` past the stale window; requeue those too.
3. Find timed-out jobs with `job_executions` rows stuck in `status = 'running'` for the topic.