Judges
LLM-as-a-judge in Stratix — building, running, and optimizing judges with GEPA.
A judge is an LLM that grades dimensions of an output — helpfulness, faithfulness, tone, safety, correctness. Judges are versioned, tunable against labeled examples (GEPA), and can evaluate either evaluation rows or traces directly.
Judges and scorers share the LLM-evaluation surface (both are model + prompt). They're separated by lifecycle:
Versioning
Immutable
Versioned with execution history
Where it runs
Inside an evaluation run
Standalone, against traces or evaluations
Optimization
n/a
GEPA-tunable against labeled examples
Anatomy
Every judge has:
Name and description
Output type — binary (pass/fail), score (e.g., 1-5), or labeled (multi-class)
Judging model — the LLM that runs the rubric
Rubric — the prompt (the judge's
evaluation_goal)Versions — every rubric edit creates a new version; execution history is recorded per version
Labeled examples (optional) — used by GEPA optimization to tune the rubric
When to use a judge (vs a scorer)
The rubric is stable and you won't iterate on it with labels
You expect to tune the rubric over time with labeled examples
You're applying it across many benchmarks as part of evaluation runs
You're evaluating traces (live or imported)
You don't need version history
You need version history for audit / rollback
You don't need GEPA tuning
You need GEPA tuning to push agreement-with-humans up
For deterministic dimensions (exact match, regex, JSON schema validity, Flesch-Kincaid grade, fairness math) use a code grader — these don't fit either the scorer or judge surface; they run as separate deterministic checks in the evaluation runtime.
Judge model selection
The judging model is a knob. Stronger models grade more reliably but cost more. Most rubrics work fine with a balanced choice; only use the strongest model when the dimension is genuinely subtle.
Judge optimization (GEPA)
GEPA tunes the judge's rubric prompt against a labeled ground-truth set. Iterate prompt variations, pick the variation with the highest agreement rate, repeat.
What GEPA actually does (algorithm sketch)
GEPA is an evolutionary prompt search, not a hand-tuning helper. Each iteration:
Score the current rubric against the labeled set; record per-example agreement.
Identify systematic disagreements — examples where the judge's verdict consistently misses the human label.
Generate candidate rubric variations targeting those disagreements (a meta-LLM proposes rubric edits).
Score each candidate against the same labeled set.
Promote the highest-scoring candidate.
Stop when score plateaus across a window of iterations or the iteration budget is reached.
Convergence typically happens within 15-25 iterations on a 30-100-example labeled set. The optimized rubric is stored as a versioned artifact alongside the original — you can roll back if it regresses on a held-out set. GEPA does not retrain the judging LLM; only the prompt changes. That keeps optimization cheap (tens of dollars rather than tens of thousands) and reproducible (the rubric is a string, not a model checkpoint).
Why GEPA matters
Out-of-the-box LLM judges agree with humans about 60-70% of the time on subtle dimensions — usable but flaky. GEPA-optimized judges typically reach 85-95% agreement. The gap matters when you're using judges as CI gates.
When to GEPA-optimize
You have ≥30 labeled examples
The judge is being used in a CI gate or in continuous evaluation
Out-of-the-box agreement isn't good enough for your bar
When NOT to GEPA-optimize
You're prototyping — eyeballing 5 outputs is fine
You don't have labels — without labels GEPA has nothing to optimize against
The dimension is well-served by a code grader (deterministic match, schema, math)
System judges
LayerLens ships system judges as starting points: helpfulness, faithfulness, safety, tone, brevity, structured-output validity. Clone and customize for your team.
Common judge dimensions
Helpfulness — does the response advance the user's goal?
Faithfulness — does the response ground its claims in retrieved or supplied context?
Safety — does the response avoid harmful content?
Tone — does the response match the desired tone?
Brevity — is the response appropriately concise?
Structured-output validity — for non-JSON-schema output formats, did the model follow them?
Where to next
Last updated
Was this helpful?