Pattern: citizen-services chatbot

Government / public sector — citizen-services chatbot pattern. Policy accuracy against versioned source-of-truth, plain-language reading-level, multilingual parity.

A state or federal agency operates an AI chatbot that helps citizens find services, check eligibility, and complete applications (benefits, permits, tax forms, licenses). The agency's source-of-truth policy documents update on their own cadence — and a stale chatbot answer can deny benefits citizens are entitled to.

This pattern shows how to evaluate citizen-services chatbots for policy accuracy, accessibility, and equity.

What's at stake

Risk dimension	Magnitude	Framework
Eligibility-misstatement civil-rights exposure	Class-action and Title VI claims	Title VI / ADA / state agency rules
Section 508 accessibility violations	Per-violation penalties	Section 508 (federal) / state equivalents
Public-trust impact from incorrect official guidance	Long-tail agency-credibility damage	Public agency-performance research
Congressional / legislative inquiry	Hearing time, executive-branch response	Public oversight records

The evaluation pattern

A policy-grounded evaluation runs against a versioned source-of-truth.

1. Custom code grader (policy-document-hash check) — every chatbot answer references the active policy document version; if the active document hash differs from the answer's referenced version, the answer is stale = fail.
2. Faithfulness judge (GEPA-tuned against ≥50 caseworker-labeled examples — scored output) — claims about eligibility and procedures are grounded in the cited policy section.
3. Reading-level scorer (Flesch-Kincaid or grade-level equivalent) — patient-facing answers must be at or below the configured grade level (commonly 6th-8th grade for citizen-services).
4. Multilingual parity scorer (custom code) — per-language accuracy must be within 5 percentage points of the primary language baseline.
5. Disclaimer-presence scorer — the chatbot must include the agency's official "this is informational, not a determination" disclaimer where required.

Don't have labels yet? See Bootstrap a judge before GEPA for the week-1 setup.

Continuous trace evaluation: sampled hourly during business hours. Policy-document-hash scorer runs on every trace (cheap and deterministic). Threshold alerts route to the agency's program managers.

Configuration in code

# Python (SDK)
from layerlens import Stratix
client = Stratix()

policy_hash = client.scorers.create_code(
 name="policy-document-hash",
 code="""
cited = output.get('policy_version_hash')
active = policy_registry.active_hash(output['program'])
result = {'passed': cited == active, 'cited': cited, 'active': active}
""",
)

reading_level = client.scorers.create_code(
 name="reading-level",
 code="result = {'passed': flesch_kincaid_grade(output['text']) <= 8}",
)

faithfulness = client.judges.create(
 name="policy-faithfulness",
 evaluation_goal="Eligibility and procedure claims must be grounded in the cited policy section.",
)

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"feature": "citizen-chatbot"}},
 scorers=[policy_hash.id, reading_level.id],
 judges=[faithfulness.id],
 schedule="hourly",
)

// TypeScript (REST)
const r = await fetch("https://stratix.layerlens.ai/api/v1/trace-evaluations", {
 method: "POST",
 headers: {
 "X-API-Key": process.env.LAYERLENS_STRATIX_API_KEY!,
 "Content-Type": "application/json",
 },
 body: JSON.stringify({
 trace_set: { tags: { feature: "citizen-chatbot" } },
 scorers: [policyHashId, readingLevelId],
 judges: [faithfulnessId],
 schedule: "hourly",
 }),
});

What you get

• Stale-policy answers detected within hours, not weeks.
• Multilingual accuracy parity is measured per language, not assumed.
• Auditor-ready evaluation history for civil-rights and accessibility audits.
• Pre-publication block prevents citizen-facing release on a regression.

Stratix capabilities used

• Custom code graders — document-hash, multilingual parity, reading-level
• Judges with GEPA optimization — policy faithfulness
• Trace evaluations — continuous sampled
• Notifications — program-manager routing

Replicate this

Get started: Cookbook: catch hallucinations is the closest runnable starter (policy faithfulness shape).

• Industry → Government and public sector
• Concept: Continuous evaluation
• Workflow: Govern