@repo/ai-agents

@repo/ai-agents houses the app's AI agents — the "V3" chart-review stack. It is organized in three tiers: generic, cross-domain building blocks (lib/, tools/), the shared per-domain chart-question stage (runners/audit-question/), and the concrete runners themselves. Everything is server-only (LangChain, OpenAI, @repo/db), so the package is guarded with server-cli-only and must never be imported from a client bundle.

Two surfaces run on it today, and the split between what they share and what they don't is the main thing to understand before changing anything:

They share the whole pipeline up to the agent call, and diverge completely after it. See The chart-question stack.

Dependencies

graph TD
  agents["@repo/ai-agents"]
  agents --> ai["@repo/ai"]
  agents --> db["@repo/db"]
  agents --> errors["@repo/errors"]
  agents --> legacy["@repo/legacy"]
  agents --> logger["@repo/logger"]
  agents --> safe["@repo/safe"]
  agents -.-> tsconfig["@repo/typescript-config"]
  agents -.-> vitest["@repo/vitest-config"]

Invocation hierarchy

The package models LLM work as four layers, each an abstract base class with a file-name suffix so every call site is discoverable by a glob:

Error / trace convention: only BaseRunner.run() opens the root span and produces a Safe. BaseAgent.invoke(), BasePrompt.invoke(), and BaseTool bodies THROW on failure and open CHILD spans (via withStepTrace) that nest under the active root; the enclosing runner captures the throw into its Safe.

Prompt TEXT + MODEL vs SCHEMA: BasePrompt and BaseAgent resolve prompt text and { model, temperature } from local override → LangSmith hub → committed snapshot (via renderHubPrompt over @repo/ai/prompts). The output SCHEMA stays in code on the subclass. Optional per-instance modelOverride wins over the resolved model.

await new DecomposeQuestionPrompt().render(input); // text only, no LLM
await new DecomposeQuestionPrompt().invoke(input, ctx);
await new DecomposeQuestionPrompt({ model: "gpt-5.1" }).invoke(input, ctx); // override
Copy

Usage

A runner extends BaseRunner<TInput, TOutput> and implements execute() plus a stable runnerName. The public run() template wraps execute() with a root LangSmith trace, structured logging, and Safe error capture, returning a uniform RunnerRun<TOutput> envelope. Run-level metadata (traceUrl, durationMs) lives OUTSIDE the Safe, so it is always available — including on failure. The domain payload (or error) is on run().result.

The base classes (BaseRunner/BaseAgent/BasePrompt/BaseTool) are package-internal building blocks imported via relative paths — the package only exports @repo/ai-agents/agents (runnable runners + agents) and @repo/ai-agents/prompts (single-shot prompts).

import { BaseRunner } from "../lib/base.runner";
import type { RunnerContext } from "../lib/result";

type GreetInput = { patientId: number; name: string };

class GreetRunner extends BaseRunner<GreetInput, { greeting: string }> {
  public readonly runnerName = "example.greet";

  // Optional: attach non-PII metadata to the trace. Merged on top of the auto-extracted
  // metadata (ids/enums/flags) and `ctx.metadata`, so it wins on key collision.
  protected traceMetadata(input: GreetInput): Record<string, unknown> {
    return { patientId: input.patientId };
  }

  protected async execute(input: GreetInput, ctx?: RunnerContext): Promise<{ greeting: string }> {
    // `ctx?.signal` is available for cooperative cancellation of model/tool calls.
    return { greeting: `Hello ${input.name}` };
  }
}

// Pass entityId/patientId as first-class trace context so AI-usage metrics attribute to them.
const run = await new GreetRunner().run({ patientId: 42, name: "Alex" }, { entityId, patientId: 42 });
console.log(run.traceUrl, run.durationMs);
if (run.result.error) {
  // handle failure
} else {
  console.log(run.result.data.greeting);
}
Copy

RunnerContext carries cross-cutting concerns and is optional, so simple request/response runners (and the agents / prompts / tools they compose) can ignore it:

• signal — cooperative cancellation, forwarded to model/tool calls.
• entityId / patientId — first-class (but optional) identifiers for per-customer AI-usage metrics. When supplied, BaseRunner stamps them onto the root LangSmith span under the canonical snake_case keys entity_id / patient_id, and withTraceCallbacks (@repo/ai/tracing) inherits those keys onto every child model/tool span — so token/cost usage aggregates by entity and patient across the whole trace tree. Pass them wherever they are in scope (ctx.user.entity_id, a job's entityId, patient.entity_id). A run with no entityId still executes but is flagged with a missing:entity-id trace tag and a warning log, so metric coverage stays queryable.
• metadata — extra non-PII key/values merged onto the root span.

Trace metadata is auto-extracted PII-safely via extractSafeMetadata from @repo/utils/keys: it lifts primitive values whose key is allowlisted (ids, pagination/sort keys, enum-ish Type/Status/Kind suffixes in camelCase or snake_case, and boolean-flag prefixes like is/has/use/should/enable) and drops free-text (e.g. question, names). BaseRunner composes this with the deployment environment before handing it to the trace.

The chart-question stack

Both runners answer a question about one patient's chart. Everything that happens before an agent sees that question is a function of the question, the patient, and the config — none of it is specific to grading — so it lives in one shared stage.

prepareChartQuestion(question, patient, config)
  │
  ├─ resolveContexts      patient context + hoisted charts / vector stores
  ├─ no-charts bailout    abstain before any LLM call is paid
  ├─ planQuestion         classify needs → ground date intent → decompose
  │                       → resolve per-criterion date windows (pass 2)
  └─ selectDocuments      narrow the chart set to what could answer it
  │
  ▼
outcome: "no-charts" | "abstain" | "ready" { contexts, plan, selection }
Copy

It returns the abstain conditions, not an abstain result — the wording of "no documents on file" belongs to whichever surface says it, and an auditor and a clinician in a text box need different sentences.

From ready, each runner does its own thing:

buildChartQuestionSystemVariables is the prompt-side counterpart to prepareChartQuestion. Put shared prompt variables there, not in buildSystemPrompt.ts — case chat previously borrowed that file's private helpers by exporting them, and the next change to the date subsystem changed their arity and broke chat's call sites.

Retrieval is deliberately not the model's decision

The agents have fetch_document, get_document_signatures, the time tools and build_timeline. They do not have search_chart or list_documents. Document selection happens in selectDocuments before the loop starts, as a hard gate with no fallback: if selection finds nothing relevant, the runner abstains rather than letting the agent go hunting. Letting the model choose what to read was a measurable source of wrong answers.

The practical consequence for chat is that "no relevant document could be selected" is a common outcome, not an edge case — which is why it has its own gate reason and its own chat-voice wording.

Gates

Three deterministic gates decide whether an answer is allowed to stand. They are the actual hallucination control, and they run for both surfaces.

Every gate returns the same shape and nothing else:

type GateDecision = {
  held: boolean;                                        // false = the gate fired
  flag: { reason: ReviewFlagReason; detail: string } | null;
};
Copy

A gate decides and says why. It never sees the answer text, never renders prose, and is never told which surface it is running for. The caller maps held onto whatever status vocabulary it owns — or has none — and renders the caveat itself:

const grounding = await applyGroundingGateStep({ isDefinitive, hasReadDocuments, enabled });
status = grounding.held ? status : "needs_review";       // chart review owns this vocabulary
answer = applyGateFlag(answer, grounding);               // auditor voice: a triageable banner
answer = applyGateFlag(answer, grounding, "chat");       // chat voice: the assistant's own hedge
Copy

applyGateFlag composes, so several gates can caveat one answer in order. The two voices live in flagForReview.ts and exist because a chart-review banner is routing instructions for a human reviewer, while in a conversation there is no worklist and nobody coming — the caveat has to change how the reader treats the answer instead of routing it somewhere.

When adding a gate: return GateDecision, add a ReviewFlagReason with both voices, and let the caller render. Don't reach for the answer text.

Config levers

AuditQuestionConfigOverrides is a deep-partial over resolveAuditQuestionConfig's zod schema (runners/audit-question/config.ts) — models, sampling, loop bounds, toolset, gates. Missing keys fall back to accuracy-first defaults, so a caller opts out rather than in. Production resolves the overrides from LaunchDarkly at the call site and passes them down; the package itself never reads a flag.

Prompt hub (dual pipeline with @repo/legacy)

Committed snapshots live in src/prompts/snapshots/. Hub names use the v2 kebab namespaces audit-question-* and case-chat-* (separate from legacy chart-review-* names). Resolution order:

1. snapshots/<name>.local.json — per-developer override (gitignored; local dev only)
2. LangSmith hub (when LANGSMITH_PROMPTS=true)
3. Committed snapshots/<name>.json — cold-start fallback (includes model)

Known gap. promptKit's listHubNames filters on the audit-question- prefix, so case-chat-answer does not appear in the CLI listing even though it carries tags: ["case-chat"]. Whether discovery should key off prefix or tags is an open decision; if it becomes tags, the existing prompts need tagging.

pnpm dev (apps/web) hydrates both @repo/legacy and @repo/ai-agents overrides. Edit a *.local.json, then pnpm prompts:migrate <name> to push to hub :latest. After promoting in the LangSmith UI, pnpm sync-prompt-snapshots <name> refreshes the committed fallback (template + model).

Updating prompts in the LangChain prompt hub

To publish local prompt edits to the LangSmith (LangChain) prompt hub, run from the repo root:

pnpm --filter @repo/ai-agents prompts:migrate [name ...]
Copy

This pushes each edited snapshots/<name>.local.json override up to the hub as the :latest revision of that prompt — it is the standard way to get a prompt change off your machine and into the hub, where it can be reviewed and promoted to :production in the LangSmith UI. Omit name to migrate every local override; pass one or more names (e.g. audit-question-decompose) to migrate specific prompts. The script loads env from apps/web/.env.local / .env, so LangSmith credentials must be configured there. Remember that promoting in the UI is not the end of the loop: follow up with pnpm --filter @repo/ai-agents sync-prompt-snapshots <name> to refresh the committed fallback snapshot so cold starts pick up the new template and model.

The CLI engine (hydrate / migrate / sync / push) lives in @repo/ai/prompts/scripts (createPromptScriptKit). This package only supplies config: snapshot dir, hub name prefix, default tags, and model-aware toPushable. Description/tags live on each snapshot JSON.

Testing

Prompt resolution is pinned to the committed snapshots for the whole suite, by two mechanisms that are both needed:

• __tests__/setup.ts sets LANGSMITH_PROMPTS=false, so no test reaches the hub and starts depending on whatever is deployed.
• localOverridesEnabled() in src/prompts/localOverride.ts excludes VITEST, so a developer's *.local.json cannot decide a test outcome. Overrides win over both the hub and the snapshot by design, so the env var alone would not have been enough — before this, a machine with local overrides was grading prompt assertions against uncommitted edits and passing for reasons CI would never reproduce.

@repo/db opens a connection at import time, so tests that pull in tool classes or repos fully replace the module (vi.mock("@repo/db", () => ({ ... }))) rather than partially mocking it with importOriginal.

Future work

Streaming runners (BaseStreamingRunner)

BaseRunner is a request/response base: run() resolves a single RunnerRun<TOutput>. Use cases that emit partial output over time — chat token streaming, artifacts that fill into a canvas live, incremental tool-call progress — need a streaming contract that the single-shot run() can't express. This is intentionally deferred; the intended design is captured here so it isn't lost:

• A sibling BaseStreamingRunner<TInput, TChunk, TOutput> alongside BaseRunner.
• Subclass implements executeStream(input, ctx?): AsyncIterable<TChunk> (the token/event stream) and reduce(chunks: TChunk[]): TOutput (fold the stream into a final value).
• stream(input, ctx?): AsyncGenerator<TChunk> — traced + logged, using langsmith traceable's native async-generator support so the root span stays open for the whole stream and flushes on completion.
• run(input, ctx?): Promise<RunnerRun<TOutput>> — consumes stream(), collects chunks, and returns reduce(chunks) in the same RunnerRun envelope as BaseRunner (so traceUrl / durationMs are always present and result is Safe<TOutput>).
• Error convention: the async iterator throws on failure (matches LangGraph .stream() and AI SDK iterator semantics). run() captures the throw into result; direct stream() consumers wrap iteration. ctx.signal abort surfaces as a thrown AbortError.
• Transport adapters (LangGraph .stream(), AI SDK toUIMessageStream) stay out of scope; TChunk is transport-agnostic.

Scripts