Legal scenarios
Legal AI evaluation scenarios — contract review, legal research, document discovery, document automation.
Legal AI's defining failure mode is the fabricated citation — Mata v. Avianca, Inc. (S.D.N.Y. 2023) is the canonical reference for sanctioned attorney misconduct stemming from AI-generated hallucinations. Every scenario below treats verifiability as the first-order property.
Contract review and analysis
Setup: AI reviews contracts to identify key clauses, flag non-standard terms, compare against a playbook, and suggest redlines.
Why evaluation is hard: material clauses can be buried in cross-references (e.g., Section 8.2(a) referencing Exhibit B, Paragraph 4) — the AI catches "unlimited liability" in plain text but misses the structural reference.
Stakes: missed unfavorable clauses (unlimited liability, auto-renewal, IP assignment); malpractice for law firms; financial exposure for in-house teams.
Eval criteria:
Clause-recall scorer — every material clause from the attorney-annotated set must surface
Cross-reference judge (GEPA-tuned ≥50 examples) — clauses defined via structural references are correctly resolved
Risk-classification judge — risk-tier matches attorney verdict
Hallucinated-clause-text rule — generated redline text must match clause-source text where quoted
Legal research assistant
Setup: AI researches case law, statutes, and regulations to support brief drafting and answer legal questions.
Why evaluation is hard: Mata v. Avianca and successor cases mean every fabricated citation in a filing is a sanctionable event — regardless of whether anyone caught it before submission.
Stakes: court sanctions per Mata; bar discipline; malpractice exposure; reputational cycle (ABA Journal coverage of every public sanction).
Eval criteria:
Citation-existence custom code grader — every cited case looks up against an authoritative legal database; any missing = fabricated. Hallucination rate must be 0% for the evaluation to pass.
Holding-accuracy judge (GEPA-tuned ≥50 attorney-labeled examples — scored output) — does the cited case actually stand for the proposition the AI claims?
Quote-accuracy scorer — quoted passages match the source verbatim
Jurisdictional-relevance judge — citations are from the right court/jurisdiction for the brief
Recency / overrule-detection scorer — never cites overruled cases (matched against a treatment database)
Discovery / e-discovery review
Setup: AI reviews documents for relevance, privilege, and responsiveness in litigation discovery.
Why evaluation is hard: the AI may correctly process body text but miss content in embedded tables and scanned attachments — relevant material in a different modality slips through.
Stakes: spoliation sanctions ($10K-$1M+); privilege waiver (catastrophic disclosure); inability to defend production proportionality.
Eval criteria:
Relevance recall scorer — must be >95% for defensibility
Privilege-detection judge (GEPA-tuned ≥50 examples) — attorney-client and work-product detection accuracy
Multimodal-coverage scorer — embedded tables, scanned attachments, image OCR captured (not skipped)
PII / personal-data detection scorer — ≥99% for GDPR / CCPA contexts
Inter-reviewer consistency scorer — Cohen's kappa or equivalent for reviewer agreement
Contract / document automation
Setup: AI drafts contracts, NDAs, and legal documents from templates + case-specific inputs.
Why evaluation is hard: template substitution errors are easy; tone, voice, and jurisdictional appropriateness require subjective judgment.
Stakes: unenforceable contracts; embarrassing template artifacts ("[CLIENT NAME]" in published doc); jurisdictional mismatches.
Eval criteria:
Template-substitution scorer (custom code) — every placeholder filled; no leftover markup
Jurisdictional-applicability judge — clauses match the governing-law jurisdiction
Voice / tone judge — matches firm's house style
Cross-reference integrity scorer — internal section references resolve correctly
Compliance / regulatory analysis
Setup: AI analyzes regulatory texts, drafts compliance memos, and identifies obligations from new regulations.
Why evaluation is hard: regulatory text is dense, hierarchical, and context-dependent. Stale regulatory interpretation (citing superseded guidance) is a frequent failure.
Eval criteria:
Regulation-currency rule — cited regulation version must match the active version (document-hash check)
Obligation-extraction scorer — every actionable obligation identified, classified by deadline / who-must-act
Faithfulness judge — every claim cites a specific regulation section
Plain-language judge (GEPA-tuned) — output translates regulatory text appropriately for the audience
See also
Industry pattern: AI research citation verification — these scenarios composed into one runnable evaluation
Last updated
Was this helpful?