# EHR sync failing

**Symptom:** a facility's census, episodes, or documents stop updating. Vendors: Kipu, Sunwave, LightningStep (API) and InSync (RPA via CloudCruise).

This is a [danger zone](../16-on-call.md#danger-zones): vendors rate-limit us and every payload is PHI. Stabilize and diagnose; don't "resync everything" to see if it helps.

## How sync runs

- The hourly `/api/cron/integrations-sync` cron prunes expired `job_executions` rows, then enqueues one `sync-integration` job per active integration.
- That fans out to `sync-census` → `sync-episode` → `sync-episode-check-triggers`, which enqueues triggered chart reviews.
- Failed `sync-episode` jobs are in `job_executions` with `status = 'failed'` and key `sync-episode:<episodeId>:docs-…`. Episodes themselves carry no failure marker, only `last_sync` timestamps.

## Narrow it down

1. **One facility or all?** All facilities on one vendor points at the vendor (outage, credentials, API change). Check their status page and Datadog for 401/403/429s from the client.
2. **One batch aborting:** a single bad record can fail a whole chunked upsert. Look for Postgres constraint errors (e.g. `23514`) in the sync logs. Past case: a Kipu same-day admit + date-only discharge violated `treatment_episodes_chrono_check`. Episode builders now skip backwards dates, so a new one means a new path is missing the guard.
3. **Data present but a column suddenly null across the board:** compare the raw vendor payloads before suspecting the parser. Past case: a Kipu call made with `phi_level: 'low'` silently omitted `level_of_care`.
4. **RPA (InSync):** Superadmin → Automation runs shows each run, with recording playback. Retry failed runs from there. The CloudCruise webhook always returns 200, so failures only show up in our logs, not as vendor retries.

## Stabilize

- `canSyncInsurances` (per entity) turns off insurance sync for one org. `triggerRevampEnabled` and `enabledProjectedDischargeTriggers` only *add* trigger types (program start, projected discharge); turning them off removes those. Admission, discharge, and UR-end triggers have no flag.
- If sync is enqueuing a burst of triggered reviews, see [queue storm](./queue-storm.md) before it becomes one.
- Leave rate limits in the vendor clients alone during an incident.

## Don't

- Don't run a full backfill to repair a few records. It overwrites fields the forward sync deliberately left alone.
- Don't edit `packages/legacy/src/integrations/` as a hotfix without a second engineer.