The rotation is 24/7. This page is what you need when something is wrong in production: how bad it is, where to look, which levers exist, and who to pull in. Symptom-specific playbooks live in on-call/.
| Role | Who | How to reach |
|---|---|---|
| Primary on-call | see rotation schedule | |
| Secondary / backup | ||
| Engineering lead | ||
| Customer contact (facility comms) | ||
| Privacy / compliance (PHI exposure) |
Hand off with a short written note in the on-call channel: open incidents, anything you changed (flags, concurrency, deleted deployments), and what to watch.
Severity decides whether you wake someone up. When unsure, pick the higher one; downgrading later is free.
| Sev | Examples | Response |
|---|---|---|
| SEV1 | App down or login broken for everyone. PHI shown to the wrong org or user. Chart reviews failing across all customers. A migration left prod half-applied. | Now, any hour. Page the secondary. Post in the on-call channel. |
| SEV2 | One integration's sync failing. One queue topic 504ing or backed up. Uploads broken for one facility. | Within the hour. Overnight, stabilize (flag off, lower concurrency) and hand the root cause to the morning. |
| SEV3 | A cosmetic bug, a slow page, one stuck run. | Next business day. File a Linear issue. |
Any suspected PHI exposure is SEV1 regardless of scale. Stop the exposure first, then contact privacy/compliance. Do not paste patient names, DOBs, MRNs, or chart text into Slack, Linear, or PRs — refer to records by ID.
service/env tags), then Datadog APM traces for the failing route or jobbable. Traces carry organization.id, patient.id, auditRun.id, and friends as span attributes, so filter by those.| Question | Source | Caveat |
|---|---|---|
| What errored? | Datadog logs | Logs go stdout → Vercel drain → Datadog with no flush in the queue handler. Under load they are sampled and dropped. |
| How many jobs ran / failed? | Datadog APM traces | Traces force-flush, so trust them over log counts. |
| Did a chart review actually happen? | audit_runs table |
The source of truth. audit_runs has no entity_id; join cases → patient. |
| Did a job fail? | job_executions.status = 'failed' |
A job killed by a 504 never reaches its catch, so it stays running. Rows past expires_at are pruned hourly; a missing row is not a signal. |
| What did the LLM do? | LangSmith | Prompt tag follows VERCEL_ENV. |
| Is the queue backed up? | Vercel dashboard → Queues | Messages are pinned to the deployment that enqueued them — see queue storm. |
| What is the browser seeing? | Datadog RUM | Off when NEXT_PUBLIC_DISABLE_DATADOG=true. |
There is no health-check endpoint. The hourly /api/cron/test cron enqueues a test job; if its trace stops appearing, the queue pipeline is down.
There is no global "stop syncing" or "stop reviews" switch. These are what exist, roughly fastest first.
| Lever | Where | Takes effect |
|---|---|---|
| LaunchDarkly flags | LaunchDarkly UI; registry in apps/web/feature-flags.d.ts |
Immediately. Operational ones: chartReviewBatchingEnabled, chartReviewBatchSize, triggeredChartReviewResumeEnabled, and per-entity canSyncInsurances. None of them stops sync or reviews outright. |
| Vercel instant rollback | Vercel → Deployments → promote previous | Immediately for requests. Does not move queued messages and does not undo migrations. |
| Delete a deployment | Vercel → Deployments | Immediately. Kills every queued message pinned to it. The emergency stop for a queue storm. |
Queue maxConcurrency / retryAfterSeconds |
apps/web/vercel.json functions triggers |
Next deploy, and only for messages that deploy enqueues. |
| Stale-run windows | AUDIT_RUNS_STALE_MINUTES, SESSION_NOTES_STALE_MINUTES env |
Next deploy. |
If LaunchDarkly is unreachable, every flag silently takes its code default (packages/feature-flags/src/server.ts). "Flags look wrong everywhere at once" can mean LaunchDarkly is down, not that someone flipped them.
forceRun, past the dedup window. Only pending questions re-run, so it is safe to repeat.failed by itself is this, not a new error.@repo/backfills CLIs: pnpm --filter @repo/backfills <script>. Dry-run first where the script supports it. Never re-run a full backfill to "fix" one record — it clobbers..github/workflows/release.yaml): prod migrations run first, then the Vercel production build.patient_charts, job_executions) are operations, not migrations. Run them CONCURRENTLY, off-peak, by hand.Don't improvise here at night. Stabilize and escalate.
packages/legacy/src/integrations/ — vendor rate limits. Never "resync everything".Within two business days, write a short blameless note: timeline, impact (orgs and run counts, no PHI), root cause, what stopped it, and follow-up Linear issues. If the playbook was wrong or missing, fixing it is part of closing the incident.
None of these exist yet; each is a gap an on-call engineer currently covers by noticing.
/api/queue/* per topic, alerting on a sustained rise.audit_runs stuck in running for longer than the stale window.test job trace.