Circle V2 API Docs
    Preparing search index...

    Class RunEvalExperimentJobbable

    Runs a LangSmith offline experiment: each dataset example (one chart-review question) is run against the eval-branch clone and scored against its graded ground truth. Stamps the git commit as the "application version" so experiments stay comparable across prompt/config changes.

    maxAttempts: 1 — an experiment is not idempotent; a retry would re-run every question and create a duplicate experiment.

    Hierarchy (View Summary)

    • Jobbable<RunEvalExperimentJobParams, void>
      • RunEvalExperimentJobbable
    Index

    Constructors

    Properties

    config: JobConfig = ...
    serviceName: "RunEvalExperimentJobbable"

    Stable, human-readable name for the service. Must be set explicitly on each subclass because this.constructor.name gets mangled by minification in production builds, which breaks tracing/observability span names.

    Accessors

    • get logger(): BaseLogger

      Returns BaseLogger

    • get name(): string

      Returns string

    Methods

    • Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns string

    • Derive the queue retentionSeconds from the effective dedup TTL. The queue overloads retentionSeconds — it's BOTH the message lifetime AND the idempotency dedup window (min(retentionSeconds, 24h)). So:

      • take the intended dedup window (ttlHours·3600), capped at 24h (the window can't exceed it);
      • but never go below the message's worst-case time-to-complete (delaySeconds + every retry), else a short TTL would expire the message mid-retry — and the queue hard-rejects delaySeconds > retentionSeconds. ttlHours <= 0 means "no dedup" → undefined: the idempotency key is dropped (see performLater), so there's no dedup window to bound — this just leaves the queue's default message lifetime in place.

      Parameters

      • ttlHours: number
      • delaySeconds: number

      Returns number | undefined

    • Extract an entityId from params for job_executions tracking. Override in subclasses when applicable.

      Parameters

      • _params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns EntityId | undefined

    • Derive a deduplication key from the params. Return null to skip deduplication (every call enqueues).

      Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns string | null

    • Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns Promise<Safe<void>>

    • Enqueue this job for async processing via Vercel Queues. Checks for deduplication via job_executions before sending.

      Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      • Optionaloptions: PerformLaterOptions

      Returns Promise<SendResult>

    • Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns Promise<void>

    • Run the job with status tracking. Used by the queue handler. Errors propagate (not caught by safe()) so the RetryHandler can decide.

      parentSpanContext (the enqueuer's span, recovered from the queue envelope) is attached as a span LINK and the run span is started as a new trace root, so each job runs in its own trace rather than collapsing a fan-out into one unbounded trace.

      Parameters

      • params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      • metadata: MessageMetadata
      • OptionalparentSpanContext: SpanContext

      Returns Promise<void>

    • Optional hook for subclasses to attach extra non-PII span attributes derived from params. The returned attributes appear on the service.${serviceName} span (and, for jobs, on job.enqueue.* and job.run.* spans too).

      Safe ID-like keys on params (e.g. patientId, templateIds, limit, cursor) are already auto-extracted via toSpanAttributes("service.input", params) — you don't need to enumerate them here. Use this hook for derived/computed attributes (counts, flags, computed names) or for fields you've explicitly verified safe but that aren't in the global allowlist. Hook output overrides auto-extracted attributes on key collision.

      Safe by default: returns {} when not overridden. Only declare keys you've confirmed are non-PII. Never include emails, names, dob, free-text, etc.

      Parameters

      • _params: {
            datasetName: string;
            experimentPrefix?: string;
            patientId?: PatientId;
            splits?: string[];
        }
        • datasetName: string

          LangSmith dataset to evaluate against, e.g. "Chart Review Run Evals".

        • OptionalexperimentPrefix?: string

          Optional prefix for the LangSmith experiment name.

        • OptionalpatientId?: PatientId

          When set, scope the experiment to this patient's examples — all review states, incoming included.

        • Optionalsplits?: string[]

          Dataset splits to evaluate, literally (union). Omit (with patientId unset) to run the golden split.

      Returns Attributes

    • Optional hook for subclasses to attach extra non-PII span attributes derived from the successful service result. Only called when run resolves successfully; on error, the span is marked via markSpanError instead.

      Safe by default: returns {} when not overridden. Only declare keys you've confirmed are non-PII (e.g. counts, status flags, computed summaries). Never include emails, names, dob, free-text, etc.

      Parameters

      • _result: void

      Returns Attributes

    • Parameters

      • params: unknown

      Returns params is {
          datasetName: string;
          experimentPrefix?: string;
          patientId?: PatientId;
          splits?: string[];
      }