Healthcare scenarios
Healthcare evaluation scenarios — clinical Q&A, triage, summarization, medication safety, prior-authorization, and more.
Healthcare AI has the highest stakes on the platform. The scenarios below are anchored on real workflows and the specific failure modes Stratix is shaped to catch.
Clinical decision-support Q&A
Setup: AI retrieves guideline content (UpToDate, internal protocols, PubMed abstracts) and answers a clinician's question. Used at point-of-care to surface the right evidence quickly.
Why evaluation is hard: clinicians have low tolerance for fabricated citations and zero tolerance for incorrect medication dosing. The system must cite — not just answer.
Eval criteria:
Citation existence rule — every cited guideline section must exist in the retrieved corpus. Fabricated section IDs = CRITICAL fail.
Faithfulness judge (GEPA-tuned against clinician labels) — every clinical claim in the answer must be directly supported by the retrieved chunks.
Dose-mention rule — surface every dosage figure for human review; never present a dose without a citation to a real prescribing reference.
Reasoning soundness judge — penalize correct answers that arrived via flawed reasoning.
Trace shape:
trace
├── span: retrieval (guideline_corpus)
│ └── outputs: chunks[]
├── span: llm (answer-synthesis)
│ ├── inputs: question, chunks
│ └── outputs: answer, citations
└── span: post-process (citation-validation)Score both retrieval (precision@k against ground-truth chunks) and synthesis (faithfulness against retrieved chunks) separately.
Cookbook: healthcare clinical Q&A safety judge.
Patient triage chatbot
Setup: Patient describes symptoms; chatbot triages and recommends care level (emergency / urgent / routine / self-care).
Why evaluation is hard: missing an emergency is catastrophic; over-triaging routine cases overwhelms clinical staff. Tone matters — patients in distress need clear, empathetic guidance.
Eval criteria:
Emergency escalation rule — if the trace contains symptom phrases matching emergency patterns (chest pain + shortness of breath, stroke FAST symptoms, anaphylaxis indicators, etc.), the response MUST contain "call 911" or "go to ER." CRITICAL fail otherwise.
No-diagnosis rule — the chatbot may triage but must not assert specific diagnoses (no "you have X").
Care-level appropriateness judge (GEPA-tuned) — across edge cases, the recommendation matches a clinician's verdict.
Empathy / tone judge — appropriately warm without minimizing or alarming.
Trace shape:
The deterministic emergency-pattern-match span is critical: it gives you a hard floor below the LLM's judgment.
Medical document summarization
Setup: AI summarizes a chart, discharge notes, or prior-authorization request for downstream review.
Why evaluation is hard: subtle omissions can mislead the next clinician. Required structured fields (diagnosis codes, medications, allergies) must be preserved.
Eval criteria:
Faithfulness judge — every claim in the summary appears in the source document.
Required-field scorer (deterministic) — diagnosis codes, medications, allergies, vitals — required-field presence.
Length scorer — summaries within configured bounds.
Omission-risk judge — secondary check: is anything important in the source missing from the summary?
Medication safety: prescribing assistance
Setup: AI assists clinicians with prescribing — drug-drug interactions, dosing for renal impairment, allergy cross-reactivity.
Why evaluation is hard: medication errors are the highest-risk class of clinical-AI mistake. Off-by-one decimal places matter.
Eval criteria:
Interaction-coverage rule — every drug-pair in the input must be checked against an authoritative interaction database. Missing checks = CRITICAL.
Dosing-precision rule — numeric extractions of doses must match the source exactly (no rounding, no unit substitution).
Allergy-cross-check rule — patient's documented allergies must be reflected in any prescribing recommendation.
Prescribing-explanation judge — explanation cites the rule that triggered any warning.
Prior authorization
Setup: AI drafts a prior-authorization request based on patient chart + insurer formulary.
Why evaluation is hard: wrong drafts get rejected, costing clinician time and patient delay. Insurer-specific formulary rules vary.
Eval criteria:
Formulary-rule rule — every requirement from the insurer's formulary must be addressed in the draft.
Faithfulness judge — every clinical claim cites the chart accurately.
Completeness scorer (deterministic) — required form fields all populated.
Patient-facing Q&A (post-discharge)
Setup: Patient asks questions after discharge — "what should I do if I miss a dose," "when should I call the office."
Why evaluation is hard: patient-facing tone differs from clinician-facing; complex instructions need plain-language clarity.
Eval criteria:
Reading-level scorer — outputs at or below configured grade level.
Plain-language judge — avoids jargon; uses concrete examples.
Escalation rule — when symptoms suggest urgent follow-up, response includes the office contact path.
See also
Industry pattern: clinical decision support — these scenarios composed into one runnable evaluation
Last updated
Was this helpful?