Faithfulness implementation

The full Faithfulness system-judge rubric — scoring bands, inputs, output contract, and tuning.

The Faithfulness system judge scores whether every factual claim in a model's OUTPUT is supported by the supplied CONTEXT. It is the workhorse judge for any RAG-pattern application. This page is the full rubric behind the System judges catalog entry.

Contract

Inputs	{{output}} (the model's answer), {{context}} (retrieved documents / source-of-truth)
Output	score ∈ 0.0–1.0 + unsupported_claims: a list of claims in the output not entailed by the context
Default model	Frontier Claude Opus-class (model_id on the judge record)
is_system	true

Rubric

The judge decomposes the output into atomic factual claims and checks each for entailment against the context, then aggregates:

Band	Score	Meaning
Fully grounded	1.0	Every claim is directly supported by the context
Mostly grounded	0.7–0.99	All material claims supported; only trivial/inferable detail unstated
Partially grounded	0.4–0.69	A material claim is unsupported but not contradicted
Weakly grounded	0.1–0.39	Multiple material claims unsupported, or one contradicts the context
Ungrounded	0.0	Core claim is fabricated or directly contradicts the context

Each unsupported claim is returned in unsupported_claims with the offending text, so a low score is actionable rather than opaque.

Tuning notes

• Penalize confident hallucinations hardest. A fluent, authoritative claim with no support in context is the most dangerous failure and should land at the bottom of the scale.
• Do not penalize appropriate refusal. If the context is genuinely insufficient and the model declines to answer (or says it cannot find the information), that is faithful behavior — it should not score low. Faithfulness measures grounding of claims made, not willingness to answer. (Whether the refusal itself was appropriate is the Refusal Quality judge's job.)
• Extraction vs. inference. Reasonable inference that a domain reader would accept as entailed by the context is grounded; a leap that introduces new facts is not.
• Pair with a deterministic check where possible. Faithfulness is subjective grounding; for citation-backed outputs, pair it with the Citation Accuracy judge and the deterministic Citation Existence code grader.

Running it

from layerlens import Stratix
client = Stratix()

faithfulness = next(
    j for j in client.judges.get_many(type="system") if j.name == "Faithfulness"
)
result = client.trace_evaluations.create(trace_id=trace_id, judge_id=faithfulness.id)
print(result.score, result.details)  # score + unsupported_claims

From the dashboard: open a trace → trace-evaluation panel → add the Faithfulness system judge.

Improving agreement

If out-of-the-box agreement with your reviewers isn't high enough, clone the judge and run GEPA optimization against ≥ 30 labeled examples spanning the score bands, then validate on a held-out slice before deploying the optimized variant.

Reference

• System judge implementation reference
• System judges — catalog
• Judges — SDK reference