Pattern: AI research citation verification

Legal — AI research citation verification pattern. Citation-existence scorer, holding-accuracy judge, per-session block on unverified output.

An AI legal research assistant answers case-law and statutory questions, drafts memo passages, and supports brief writing. The output cites cases, statutes, and regulations. A single fabricated citation in a filing is a sanctionable event — see Mata v. Avianca, Inc., No. 22-cv-1461 (S.D.N.Y. June 22, 2023).

This pattern shows how to evaluate every research-tool output so a fabricated citation can never reach a court filing.

What's at stake

Risk dimension	Magnitude	Framework
Court sanctions per Mata-style filing	Cited sanctions $5K + reputational loss	Public court orders
Bar discipline	Public reprimand to suspension; case-by-case	State bar rules of professional conduct
Malpractice exposure for filed hallucinated citations	Multi-million-dollar settlements	Public legal-malpractice filings
Firm reputation in client and ABA Journal coverage	Long-tail damage cycle	Industry trade press

The evaluation pattern

A two-stage evaluation runs on every research-tool output:

1. Custom code grader (citation existence) — every cited case, statute, or regulation is looked up against an authoritative legal database. Any citation that doesn't resolve = fabricated. Hallucination rate must be 0% for the evaluation to pass.
2. Holding-accuracy judge (GEPA-tuned against ≥50 attorney-labeled examples — scored output) — for cases that exist, does the cited case actually stand for the proposition the AI claims? Does the quoted passage match the source verbatim?

Don't have labels yet? See Bootstrap a judge before GEPA for the week-1 setup.

1. Continuous trace evaluation — every production research session is scored. Sessions with any unverified citation are flagged in-app for attorney review before any output can be acted on. The flag is a hard gate; attorneys can override only after explicit acknowledgment.
2. Compare-models — when prompt or model changes are proposed, run the existing regression set across both; pick the variant with the lower hallucination rate at equal or better holding-accuracy.

The same pattern works for statutes, regulations, and internal precedent citations.

Configuration in code

# Python (SDK)
from layerlens import Stratix
client = Stratix()

citation_scorer = client.scorers.create_code(
 name="citation-existence",
 code="""
unresolved = [c for c in extract_citations(output) if not legal_db.lookup(c)]
result = {'passed': len(unresolved) == 0, 'fabricated': unresolved}
""",
)

holding_judge = client.judges.create(
 name="holding-accuracy",
 evaluation_goal="For each cited case, does the case actually stand for the proposition the OUTPUT claims? Quoted passages must match the source verbatim.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "legal-research"}},
 scorers=[citation_scorer.id],
 judges=[holding_judge.id],
 schedule="per-session",
)

// TypeScript (REST) — same trace eval from a Node service
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "legal-research" } },
 scorers: [citationScorerId],
 judges: [holdingJudgeId],
 schedule: "per-session",
 }),
});

What you get

• Hallucination rate held at 0% on filed-output paths — the gate is a hard block, not a warning.
• Holding-accuracy judge agreement with senior associates above 90% after GEPA optimization.
• Documented evaluation evidence — the evaluation IDs and per-citation verification artifacts — to show partners, clients, and (if needed) the court that AI-assisted research operates under verification controls.
• Attorneys treat the AI as a research-acceleration tool, not a citation source. Which is what the AI should be.

Stratix capabilities used

• Custom code grader — citation-existence verification against legal databases
• Judges with GEPA optimization — holding accuracy and quote accuracy
• Trace evaluations — every production session
• Notifications — flag unverified citations for attorney review

Replicate this

• Industry → Legal
• Cookbook: catch hallucinations
• Cookbook: legal contract-review faithfulness (analogous pattern for transactional work)
• Use case: RAG evaluation