System judge implementation reference

Implementation reference for Stratix system judges — record shape, scoring contracts, and how they run.

This is the implementation-level companion to the System judges catalog. The catalog page is the "what and when"; this section is the "how": the exact record shape, the scoring contract each judge returns, and how a judge is stored and executed.

Record shape

Every system judge is a row in the platform judges collection with the same fields as a custom judge, plus the system markers:

Field	Meaning
name	Human-readable judge name (e.g. Faithfulness)
evaluation_goal	The natural-language rubric the judge LLM is instructed with
model_id	The judge model (system judges default to a frontier Claude Opus-class model)
is_system	true for LayerLens-shipped judges
organization_id	null for system judges; set for cloned/custom judges
inputs	Template variables the rubric expects (e.g. {{output}}, {{context}})

System judges are read-only records — customization is clone-then-edit, and tuning is done with GEPA optimization against your own labeled set. See the catalog page for the clone workflow.

Scoring contracts

System judges return one of three shapes, depending on the dimension:

• Scalar — a 0.0–1.0 score plus a structured detail list. Faithfulness (unsupported claims), Citation Accuracy (citation failures).
• Categorical verdict — one of a fixed enum. Refusal Quality (appropriate_answer / appropriate_refusal / over_refusal / under_refusal), Hallucination Detector severity (none / minor / material / dangerous).
• Sub-scored — an overall score with named sub-scores. Tone & Register (warmth / formality / brevity / jargon), Explanation Quality (clarity / completeness / traceability).

The scalar-with-detail contract is documented end to end in the worked example:

• Faithfulness — implementation reference

All system judges — implementation contracts

Every system judge in the catalog and its implementation-level contract (template inputs + return shape). All share the same record fields (name, evaluation_goal, model_id, is_system: true) and run on the same LLM-as-Judge engine.

General system judges

Judge	Inputs	Output contract
Faithfulness	{{output}}, {{context}}	scalar 0.0–1.0 + unsupported_claims[]
Hallucination Detector	{{output}}, {{context}}?	severity none / minor / material / dangerous + flagged claims
Refusal Quality	{{prompt}}, {{output}}, {{context}}?, {{scope}}?	verdict appropriate_answer / appropriate_refusal / over_refusal / under_refusal
Citation Accuracy	{{output}}, {{context}}	scalar 0.0–1.0 + failures not_in_source / misquoted / wrong_section
Tone & Register	{{output}}, {{expected_register}}	overall + sub-scores: warmth / formality / brevity / jargon
De-escalation Quality	{{context}}, {{output}}	overall + acknowledgement / pathway / tone (failure sub-booleans)
Plain-language Quality	{{output}}, {{audience}}	scalar + concept-unpacking / actionability / jargon (pair with the Flesch-Kincaid code grader)
Explanation Quality	{{decision}}, {{output}}, {{reviewer_profile}}	sub-scores: clarity / completeness / traceability
Multilingual Parity	{{source_output}}, {{output}}, {{target_language}}, {{glossary}}?	parity scalar (accuracy / tone / required disclosures preserved)
Editorial Judgment	{{context}}, {{output}}	framing / attribution / headline-alignment + inserted-bias flag

Industry-specialty system judges

Same record fields and clone-and-tune workflow; these target high-stakes verticals.

Judge	Industry	What the verdict asserts
Contract Clause Interpretation	Legal, real estate	Interpretation matches the clause's plain meaning + governing law
Mata Citation Verifier	Legal	Each filed citation exists, is good law, and supports the claim
Privilege Leak Detector	Legal	Output does not leak attorney-client privileged content
Clinical Reasoning Soundness	Healthcare	Differential diagnosis appropriate; red-flag features identified
Medical-coding Justification	Healthcare	ICD-10-CM / CPT / HCPCS codes supported by documentation; no unbundling/upcoding
Switching-step Faithfulness	Energy & utilities	Grid switching sequence matches approved procedure + safety preconditions
Network Mitigation Rationale	Telecom, energy	Recommended mitigation is grounded in the runbook
Fare-rule Paraphrase Quality	Travel	Paraphrase preserves conditions, dollar amounts, and time windows
Steering-language Detector	Real estate	Listing copy does not steer on a protected-class basis
Pedagogical Scaffolding	Education	Tutor guides the learner rather than giving the answer outright

For the full worked rubric (bands, tuning, run example), see Faithfulness — implementation reference; the other scalars follow the same band structure, and the categorical judges return the enum shown above.

How a judge runs

A judge is executed by the platform's LLM-as-Judge engine: the evaluation_goal rubric is rendered with the invocation's inputs, sent to the judge model_id, and the structured verdict is persisted against the trace or evaluation. It is the same engine whether the judge is a system judge or one of your own.

• Dashboard: Judges → filter Type: System → open a judge to read its rubric.
• SDK / REST: list and run judges programmatically.

from layerlens import Stratix
client = Stratix()

# Discover system judges
for j in client.judges.get_many(type="system"):
    print(j.id, j.name, j.model_id)

# Run one against a trace
result = client.trace_evaluations.create(trace_id=trace_id, judge_id=judge_id)

Reference

• Judges — API reference
• Judges — SDK reference
• System judges — catalog
• Judge Optimization (GEPA)