# Bad release or migration

**Symptom:** errors start right after a GitHub Release, or the `release.yaml` workflow fails partway.

## How a release runs

`release.yaml`, triggered by publishing a `v*` release:

1. Update release notes.
2. **Migrate prod Supabase** (`pnpm migrate` in `packages/db`).
3. Migrate the eval DB.
4. `vercel build --prod`, upload sourcemaps to Datadog, `vercel deploy --prebuilt --target=production`.

If step 2 fails, the new code never deploys, but **prod may be left partly migrated**. Drizzle runs the migration batch in one transaction, so a failed migration usually rolls back cleanly. Hand-applied changes are what break that assumption.

## App is broken, migration succeeded

1. Vercel → Deployments → promote the previous production deployment (instant rollback).
2. Safe if that release's migrations were additive (new tables, columns, indexes). **Unsafe if a column was renamed, dropped, or made non-null**: the old code will fail against the new schema. Then fix forward.
3. Rollback doesn't move queued messages. Jobs enqueued by the bad deployment still run on it until it is deleted — see [queue storm](./queue-storm.md).

## Migration failed

1. Read the workflow log for the failing statement. Don't re-run the workflow blind.
2. If someone hand-applied part of the change to prod, the tracked migration must be made idempotent (`IF NOT EXISTS`) before it can re-run.
3. Index builds on large tables (`patient_charts` is ~19 GB) don't belong in tracked migrations: they run in a transaction, so not `CONCURRENTLY`. Build them by hand, `CONCURRENTLY`, off-peak. Even a concurrent build can saturate disk I/O, so watch Supabase metrics while it runs.
4. A failed migration can break features with no ERROR log. In v0.1.149 uploads silently failed to create a chart row. Check the features the migration touches, not just the error rate.

## Don't

- Don't write down migrations or edit an applied migration file.
- Don't approve migrations touching RLS during an incident without a second engineer.