> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-4.md). # Healthcare evaluation patterns A field guide to which Stratix primitive to reach for in clinical-AI scenarios. The pattern for healthcare differs from other industries because of the asymmetric cost of errors: **a deterministic rule that catches one bad output is worth more than a judge that grades fifty good ones.** ## Rule of thumb: 70 / 20 / 10 For a typical healthcare evaluation: * **\~70% deterministic rules** — the bulk of safety enforcement * **\~20% natural-language assertions** — fuzzy correctness (cited correctly? appropriate care level?) * **\~10% LLM judges** — residual subjective dimensions (tone, plain-language quality) Inverting this ratio (judge-heavy) is the most common mistake. ## Pattern table | Dimension | Approach | Notes | | ----------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------ | | Clinical-claim faithfulness | LLM judge, GEPA-optimized | Tune against ≥50 clinician-labeled examples; hold out ≥20 for validation | | Citation existence | Deterministic rule | Cited section IDs must resolve to real chunks | | Dose-recommendation surfacing | Deterministic rule + numeric scorer | Extract every dose mention; require exact match to source | | Drug-interaction coverage | Deterministic rule | Every drug pair in input must trigger a check | | Emergency escalation | Deterministic rule (regex / classifier) | Hard floor below LLM judgment | | Reasoning soundness | LLM judge, GEPA-optimized | Penalize correct-answer-by-flawed-reasoning | | PHI redaction | Deterministic rule + regex | Block unredacted PHI in any logged output | | Care-level appropriateness | LLM judge | Triage scenarios | | Empathy / patient-facing tone | LLM judge | Use sparingly | | Reading level | Deterministic scorer (Flesch-Kincaid or similar) | Required for patient-facing | | Required-field presence | Custom code grader | Diagnoses, medications, allergies, vitals | ## Span-level rules Healthcare agents often need rules at the span level — examples: * **Retrieval span:** "the retrieval span must not return content from outside the approved guideline corpus." * **LLM span:** "the prescribing span must include a citation to a prescribing reference for any dose mentioned." * **Tool span:** "the EHR write-back span may only fire for the patient ID in `inputs`." Stratix's span-level scoring makes these expressible. See [span-level scoring recipe](/more-in-this-section-6/span-level-scoring.md). ## Trace-set curation A healthcare trace set should over-sample edge cases: * Allergy combinations * Pediatric / geriatric weight-based dosing * Pregnancy / lactation contraindications * Renal / hepatic impairment dosing * Crisis scenarios (suicidal ideation, anaphylaxis, MI symptoms) * Multilingual presentations * Low-health-literacy phrasings A 50-trace curated set tilted toward edge cases catches more failures than a 500-trace random sample. ## GEPA labeling for clinical judges Clinicians label inconsistently across specialties; calibrate before optimizing. 1. Have ≥3 clinicians label the same 30 examples 2. Compute inter-rater agreement (Cohen's kappa or similar) 3. Discuss disagreements; converge on a rubric 4. Have those clinicians label 100+ examples with the converged rubric 5. GEPA-optimize the judge against the consensus labels Skipping the calibration step leaves you tuning a judge to noise. ## Continuous evaluation cadence Production healthcare AI should run trace evaluations: * **Per-PR** on a curated 50-trace gating set * **Daily** on a sampled production slice * **Weekly** full review with clinical leadership of any drift ## See also * [Scenarios](/more-in-this-section-2/scenarios-4.md) * [Compliance](/more-in-this-section-2/compliance-4.md) * [Cookbook: clinical Q\&A](/more-in-this-section-2/industry-healthcare-clinical-qa.md) * [Tutorial 5: GEPA optimize](/9.-improve-tune-the-system/05-gepa-optimize.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-4.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.