Healthcare evaluation patterns

Healthcare evaluation patterns — common scorer, rule, and judge combinations.

A field guide to which Stratix primitive to reach for in clinical-AI scenarios. The pattern for healthcare differs from other industries because of the asymmetric cost of errors: a deterministic rule that catches one bad output is worth more than a judge that grades fifty good ones.

Rule of thumb: 70 / 20 / 10

For a typical healthcare evaluation:

~70% deterministic rules — the bulk of safety enforcement
~20% natural-language assertions — fuzzy correctness (cited correctly? appropriate care level?)
~10% LLM judges — residual subjective dimensions (tone, plain-language quality)

Inverting this ratio (judge-heavy) is the most common mistake.

Pattern table

Dimension

Approach

Notes

Clinical-claim faithfulness

LLM judge, GEPA-optimized

Tune against ≥50 clinician-labeled examples; hold out ≥20 for validation

Citation existence

Deterministic rule

Cited section IDs must resolve to real chunks

Dose-recommendation surfacing

Deterministic rule + numeric scorer

Extract every dose mention; require exact match to source

Drug-interaction coverage

Deterministic rule

Every drug pair in input must trigger a check

Emergency escalation

Deterministic rule (regex / classifier)

Hard floor below LLM judgment

Reasoning soundness

LLM judge, GEPA-optimized

Penalize correct-answer-by-flawed-reasoning

PHI redaction

Deterministic rule + regex

Block unredacted PHI in any logged output

Care-level appropriateness

LLM judge

Triage scenarios

Empathy / patient-facing tone

LLM judge

Use sparingly

Reading level

Deterministic scorer (Flesch-Kincaid or similar)

Required for patient-facing

Required-field presence

Custom code grader

Diagnoses, medications, allergies, vitals

Span-level rules

Healthcare agents often need rules at the span level — examples:

Retrieval span: "the retrieval span must not return content from outside the approved guideline corpus."
LLM span: "the prescribing span must include a citation to a prescribing reference for any dose mentioned."
Tool span: "the EHR write-back span may only fire for the patient ID in inputs."

Stratix's span-level scoring makes these expressible. See span-level scoring recipe.

Trace-set curation

A healthcare trace set should over-sample edge cases:

Allergy combinations
Pediatric / geriatric weight-based dosing
Pregnancy / lactation contraindications
Renal / hepatic impairment dosing
Crisis scenarios (suicidal ideation, anaphylaxis, MI symptoms)
Multilingual presentations
Low-health-literacy phrasings

A 50-trace curated set tilted toward edge cases catches more failures than a 500-trace random sample.

GEPA labeling for clinical judges

Clinicians label inconsistently across specialties; calibrate before optimizing.

Have ≥3 clinicians label the same 30 examples
Compute inter-rater agreement (Cohen's kappa or similar)
Discuss disagreements; converge on a rubric
Have those clinicians label 100+ examples with the converged rubric
GEPA-optimize the judge against the consensus labels

Skipping the calibration step leaves you tuning a judge to noise.

Continuous evaluation cadence

Production healthcare AI should run trace evaluations:

Per-PR on a curated 50-trace gating set
Daily on a sampled production slice
Weekly full review with clinical leadership of any drift

Good afternoon