> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-2/scenarios-6.md).

# Legal scenarios

Legal AI evaluation scenarios — contract review, legal research, document discovery, document automation.

Legal AI's defining failure mode is the **fabricated citation** — *Mata v. Avianca, Inc.* (S.D.N.Y. 2023) is the canonical reference for sanctioned attorney misconduct stemming from AI-generated hallucinations. Every scenario below treats verifiability as the first-order property.

## Contract review and analysis <a href="#contract-review" id="contract-review"></a>

**Setup:** AI reviews contracts to identify key clauses, flag non-standard terms, compare against a playbook, and suggest redlines.

**Why evaluation is hard:** material clauses can be buried in **cross-references** (e.g., Section 8.2(a) referencing Exhibit B, Paragraph 4) — the AI catches "unlimited liability" in plain text but misses the structural reference.

**Stakes:** missed unfavorable clauses (unlimited liability, auto-renewal, IP assignment); malpractice for law firms; financial exposure for in-house teams.

**Eval criteria:**

* Clause-recall scorer — every material clause from the attorney-annotated set must surface
* Cross-reference judge (GEPA-tuned ≥50 examples) — clauses defined via structural references are correctly resolved
* Risk-classification judge — risk-tier matches attorney verdict
* Hallucinated-clause-text rule — generated redline text must match clause-source text where quoted

## Legal research assistant <a href="#legal-research" id="legal-research"></a>

**Setup:** AI researches case law, statutes, and regulations to support brief drafting and answer legal questions.

**Why evaluation is hard:** *Mata v. Avianca* and successor cases mean **every fabricated citation in a filing is a sanctionable event** — regardless of whether anyone caught it before submission.

**Stakes:** court sanctions per *Mata*; bar discipline; malpractice exposure; reputational cycle (ABA Journal coverage of every public sanction).

**Eval criteria:**

* Citation-existence custom code grader — every cited case looks up against an authoritative legal database; any missing = fabricated. Hallucination rate must be 0% for the evaluation to pass.
* Holding-accuracy judge (GEPA-tuned ≥50 attorney-labeled examples — scored output) — does the cited case actually stand for the proposition the AI claims?
* Quote-accuracy scorer — quoted passages match the source verbatim
* Jurisdictional-relevance judge — citations are from the right court/jurisdiction for the brief
* Recency / overrule-detection scorer — never cites overruled cases (matched against a treatment database)

## Discovery / e-discovery review <a href="#discovery" id="discovery"></a>

**Setup:** AI reviews documents for relevance, privilege, and responsiveness in litigation discovery.

**Why evaluation is hard:** the AI may correctly process body text but miss content in **embedded tables and scanned attachments** — relevant material in a different modality slips through.

**Stakes:** spoliation sanctions ($10K-$1M+); privilege waiver (catastrophic disclosure); inability to defend production proportionality.

**Eval criteria:**

* Relevance recall scorer — must be >95% for defensibility
* Privilege-detection judge (GEPA-tuned ≥50 examples) — attorney-client and work-product detection accuracy
* Multimodal-coverage scorer — embedded tables, scanned attachments, image OCR captured (not skipped)
* PII / personal-data detection scorer — ≥99% for GDPR / CCPA contexts
* Inter-reviewer consistency scorer — Cohen's kappa or equivalent for reviewer agreement

## Contract / document automation <a href="#document-automation" id="document-automation"></a>

**Setup:** AI drafts contracts, NDAs, and legal documents from templates + case-specific inputs.

**Why evaluation is hard:** template substitution errors are easy; tone, voice, and jurisdictional appropriateness require subjective judgment.

**Stakes:** unenforceable contracts; embarrassing template artifacts ("\[CLIENT NAME]" in published doc); jurisdictional mismatches.

**Eval criteria:**

* Template-substitution scorer (custom code) — every placeholder filled; no leftover markup
* Jurisdictional-applicability judge — clauses match the governing-law jurisdiction
* Voice / tone judge — matches firm's house style
* Cross-reference integrity scorer — internal section references resolve correctly

## Compliance / regulatory analysis <a href="#compliance-analysis" id="compliance-analysis"></a>

**Setup:** AI analyzes regulatory texts, drafts compliance memos, and identifies obligations from new regulations.

**Why evaluation is hard:** regulatory text is dense, hierarchical, and context-dependent. Stale regulatory interpretation (citing superseded guidance) is a frequent failure.

**Eval criteria:**

* Regulation-currency rule — cited regulation version must match the active version (document-hash check)
* Obligation-extraction scorer — every actionable obligation identified, classified by deadline / who-must-act
* Faithfulness judge — every claim cites a specific regulation section
* Plain-language judge (GEPA-tuned) — output translates regulatory text appropriately for the audience

## See also

* [Industry pattern: AI research citation verification](/4.2-industry-use-cases/pattern-2.md) — these scenarios composed into one runnable evaluation
* [Eval patterns](/more-in-this-section-2/eval-patterns-6.md) | [Compliance](/more-in-this-section-2/compliance-6.md)
* [SDK sample: legal\_contracts.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/legal_contracts.py)
* [SDK sample: legal\_research.py](https://github.com/layerlens/stratix-python/blob/main/samples/industry/legal_research.py)