> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-6.md). # Legal evaluation patterns Legal AI's primary evaluation cost is **verification**, not subjective grading. The pattern shape inverts what works in customer-service AI — deterministic checks dominate, judges fill the residual. ## Rule of thumb: 65 / 25 / 10 For a typical legal evaluation: * **\~65% deterministic rules / scorers** — citation existence, jurisdictional applicability, template substitution, privilege detection, document-hash checks * **\~25% natural-language assertions** — clause coverage, obligation extraction, multimodal coverage * **\~10% LLM judges** — holding accuracy, voice/tone, ambiguity resolution The judge ratio is low because the cost of judge errors is asymmetric — one missed fabricated citation is a sanctionable event. ## Pattern table | Dimension | Approach | Notes | | --------------------------------- | ------------------------------------------ | --------------------------------------------------- | | Citation existence | Custom code grader + legal database lookup | Hallucination rate must be 0% on filed-output paths | | Holding accuracy | LLM judge, GEPA-tuned | ≥50 attorney-labeled examples | | Quote accuracy | Custom code grader | Verbatim match against source | | Jurisdictional relevance | LLM judge | Court / jurisdiction matches the brief | | Recency / overrule detection | Custom code grader + treatment database | Never cite overruled cases | | Clause recall | Custom code grader | Against attorney-annotated set | | Cross-reference integrity | LLM judge, GEPA-tuned | Structurally referenced clauses correctly resolved | | Risk-tier classification | LLM judge | Matches attorney's risk verdict | | Privilege detection | LLM judge, GEPA-tuned | Attorney-client + work-product | | Multimodal coverage | Custom code grader | Embedded tables, scanned attachments, image OCR | | PII / personal-data detection | Regex + classifier | GDPR / CCPA contexts | | Document-hash regulation currency | Custom code grader | Cited regulation version matches active version | | Template substitution | Custom code grader | No leftover placeholders | | Voice / tone consistency | LLM judge | Firm house-style adherence | ## Span-level rules * **Retrieval span:** every chunk returned must be from an approved legal database (no public-web bleed when researching against case law). * **Synthesis span:** every citation in the output must trace to a chunk in the retrieval span. * **Post-process span:** citation-validation rule fires after synthesis; output is gated on this span passing. ## GEPA labeling for legal judges Holding-accuracy and privilege-detection judges benefit from GEPA — but legal labeling needs careful structure. 1. **Multi-attorney labels.** Have ≥3 senior attorneys label the same examples; compute Cohen's kappa. 2. **Stratify by practice area.** Holding accuracy in IP differs from contracts differs from criminal — separate sets per practice area. 3. **Hold out 20% per practice area.** Validation scoring per area, not pooled. 4. **Re-optimize annually** as case law evolves. ## Trace-set curation Over-sample edge cases: * Cases with cross-references (8.2(a) → Exhibit B, etc.) * Documents with relevant content in scanned attachments only * Old cases that have been overruled / abrogated * Multi-jurisdictional questions (federal + state, multiple state laws) * Documents in non-English languages * Cases involving specific protected classes (privilege concerns) ## Continuous evaluation cadence * **Per-PR** on the regression suite (citation existence + privilege detection + clause recall) * **Pre-publication block** — every research-tool output must pass citation existence before any attorney can act on it (zero tolerance) * **Daily** sample of production output for holding-accuracy and quote-accuracy * **Quarterly** — validate against new sanctioned cases (any new *Mata*-class ruling adds to the regression set) ## Guard rails for filed-output paths The platform-usage pattern for filed legal output: 1. AI generates draft 2. Stratix evaluation runs — citation existence, holding accuracy, quote accuracy 3. **Hard gate: any unverified citation blocks the output** 4. Attorney reviews; can override only with explicit acknowledgment 5. Final filing recorded with the evaluation ID for the audit trail ## See also * [Scenarios](/more-in-this-section-2/scenarios-6.md) | [Compliance](/more-in-this-section-2/compliance-6.md) * [Industry pattern: AI research citation verification](/4.2-industry-use-cases/pattern-2.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-6.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.