> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-2/eval-patterns-6.md).

# Legal evaluation patterns

Legal evaluation patterns — primitive ratios, dimensions, citation verification depth, GEPA labeling.

Legal AI's primary evaluation cost is **verification**, not subjective grading. The pattern shape inverts what works in customer-service AI — deterministic checks dominate, judges fill the residual.

## Rule of thumb: 65 / 25 / 10

For a typical legal evaluation:

* **\~65% deterministic rules / scorers** — citation existence, jurisdictional applicability, template substitution, privilege detection, document-hash checks
* **\~25% natural-language assertions** — clause coverage, obligation extraction, multimodal coverage
* **\~10% LLM judges** — holding accuracy, voice/tone, ambiguity resolution

The judge ratio is low because the cost of judge errors is asymmetric — one missed fabricated citation is a sanctionable event.

## Pattern table

| Dimension                         | Approach                                   | Notes                                               |
| --------------------------------- | ------------------------------------------ | --------------------------------------------------- |
| Citation existence                | Custom code grader + legal database lookup | Hallucination rate must be 0% on filed-output paths |
| Holding accuracy                  | LLM judge, GEPA-tuned                      | ≥50 attorney-labeled examples                       |
| Quote accuracy                    | Custom code grader                         | Verbatim match against source                       |
| Jurisdictional relevance          | LLM judge                                  | Court / jurisdiction matches the brief              |
| Recency / overrule detection      | Custom code grader + treatment database    | Never cite overruled cases                          |
| Clause recall                     | Custom code grader                         | Against attorney-annotated set                      |
| Cross-reference integrity         | LLM judge, GEPA-tuned                      | Structurally referenced clauses correctly resolved  |
| Risk-tier classification          | LLM judge                                  | Matches attorney's risk verdict                     |
| Privilege detection               | LLM judge, GEPA-tuned                      | Attorney-client + work-product                      |
| Multimodal coverage               | Custom code grader                         | Embedded tables, scanned attachments, image OCR     |
| PII / personal-data detection     | Regex + classifier                         | GDPR / CCPA contexts                                |
| Document-hash regulation currency | Custom code grader                         | Cited regulation version matches active version     |
| Template substitution             | Custom code grader                         | No leftover placeholders                            |
| Voice / tone consistency          | LLM judge                                  | Firm house-style adherence                          |

## Span-level rules

* **Retrieval span:** every chunk returned must be from an approved legal database (no public-web bleed when researching against case law).
* **Synthesis span:** every citation in the output must trace to a chunk in the retrieval span.
* **Post-process span:** citation-validation rule fires after synthesis; output is gated on this span passing.

## GEPA labeling for legal judges

Holding-accuracy and privilege-detection judges benefit from GEPA — but legal labeling needs careful structure.

1. **Multi-attorney labels.** Have ≥3 senior attorneys label the same examples; compute Cohen's kappa.
2. **Stratify by practice area.** Holding accuracy in IP differs from contracts differs from criminal — separate sets per practice area.
3. **Hold out 20% per practice area.** Validation scoring per area, not pooled.
4. **Re-optimize annually** as case law evolves.

## Trace-set curation

Over-sample edge cases:

* Cases with cross-references (8.2(a) → Exhibit B, etc.)
* Documents with relevant content in scanned attachments only
* Old cases that have been overruled / abrogated
* Multi-jurisdictional questions (federal + state, multiple state laws)
* Documents in non-English languages
* Cases involving specific protected classes (privilege concerns)

## Continuous evaluation cadence

* **Per-PR** on the regression suite (citation existence + privilege detection + clause recall)
* **Pre-publication block** — every research-tool output must pass citation existence before any attorney can act on it (zero tolerance)
* **Daily** sample of production output for holding-accuracy and quote-accuracy
* **Quarterly** — validate against new sanctioned cases (any new *Mata*-class ruling adds to the regression set)

## Guard rails for filed-output paths

The platform-usage pattern for filed legal output:

1. AI generates draft
2. Stratix evaluation runs — citation existence, holding accuracy, quote accuracy
3. **Hard gate: any unverified citation blocks the output**
4. Attorney reviews; can override only with explicit acknowledgment
5. Final filing recorded with the evaluation ID for the audit trail

## See also

* [Scenarios](/more-in-this-section-2/scenarios-6.md) | [Compliance](/more-in-this-section-2/compliance-6.md)
* [Industry pattern: AI research citation verification](/4.2-industry-use-cases/pattern-2.md)