Legal evaluation patterns

Legal evaluation patterns — primitive ratios, dimensions, citation verification depth, GEPA labeling.

Legal AI's primary evaluation cost is verification, not subjective grading. The pattern shape inverts what works in customer-service AI — deterministic checks dominate, judges fill the residual.

Rule of thumb: 65 / 25 / 10

For a typical legal evaluation:

• ~65% deterministic rules / scorers — citation existence, jurisdictional applicability, template substitution, privilege detection, document-hash checks
• ~25% natural-language assertions — clause coverage, obligation extraction, multimodal coverage
• ~10% LLM judges — holding accuracy, voice/tone, ambiguity resolution

The judge ratio is low because the cost of judge errors is asymmetric — one missed fabricated citation is a sanctionable event.

Pattern table

Dimension	Approach	Notes
Citation existence	Custom code grader + legal database lookup	Hallucination rate must be 0% on filed-output paths
Holding accuracy	LLM judge, GEPA-tuned	≥50 attorney-labeled examples
Quote accuracy	Custom code grader	Verbatim match against source
Jurisdictional relevance	LLM judge	Court / jurisdiction matches the brief
Recency / overrule detection	Custom code grader + treatment database	Never cite overruled cases
Clause recall	Custom code grader	Against attorney-annotated set
Cross-reference integrity	LLM judge, GEPA-tuned	Structurally referenced clauses correctly resolved
Risk-tier classification	LLM judge	Matches attorney's risk verdict
Privilege detection	LLM judge, GEPA-tuned	Attorney-client + work-product
Multimodal coverage	Custom code grader	Embedded tables, scanned attachments, image OCR
PII / personal-data detection	Regex + classifier	GDPR / CCPA contexts
Document-hash regulation currency	Custom code grader	Cited regulation version matches active version
Template substitution	Custom code grader	No leftover placeholders
Voice / tone consistency	LLM judge	Firm house-style adherence

Span-level rules

• Retrieval span: every chunk returned must be from an approved legal database (no public-web bleed when researching against case law).
• Synthesis span: every citation in the output must trace to a chunk in the retrieval span.
• Post-process span: citation-validation rule fires after synthesis; output is gated on this span passing.

GEPA labeling for legal judges

Holding-accuracy and privilege-detection judges benefit from GEPA — but legal labeling needs careful structure.

1. Multi-attorney labels. Have ≥3 senior attorneys label the same examples; compute Cohen's kappa.
2. Stratify by practice area. Holding accuracy in IP differs from contracts differs from criminal — separate sets per practice area.
3. Hold out 20% per practice area. Validation scoring per area, not pooled.
4. Re-optimize annually as case law evolves.

Trace-set curation

Over-sample edge cases:

• Cases with cross-references (8.2(a) → Exhibit B, etc.)
• Documents with relevant content in scanned attachments only
• Old cases that have been overruled / abrogated
• Multi-jurisdictional questions (federal + state, multiple state laws)
• Documents in non-English languages
• Cases involving specific protected classes (privilege concerns)

Continuous evaluation cadence

• Per-PR on the regression suite (citation existence + privilege detection + clause recall)
• Pre-publication block — every research-tool output must pass citation existence before any attorney can act on it (zero tolerance)
• Daily sample of production output for holding-accuracy and quote-accuracy
• Quarterly — validate against new sanctioned cases (any new Mata-class ruling adds to the regression set)

Guard rails for filed-output paths

The platform-usage pattern for filed legal output:

1. AI generates draft
2. Stratix evaluation runs — citation existence, holding accuracy, quote accuracy
3. Hard gate: any unverified citation blocks the output
4. Attorney reviews; can override only with explicit acknowledgment
5. Final filing recorded with the evaluation ID for the audit trail

See also

• Scenarios | Compliance
• Industry pattern: AI research citation verification