Judges

Build, test, and manage LLM-as-a-judge graders in Stratix Premium.

Available in Stratix Premium. This surface is part of the logged-in workspace at stratix.layerlens.ai. Stratix Public users can browse the catalog but cannot use this feature.

A judge is an LLM that grades an output along a subjective dimension — helpfulness, faithfulness, tone, safety. Stratix Premium ships with system judges, lets you build custom judges, and tunes them with GEPA.

URL: stratix.layerlens.ai/dashboard/agent-evaluation/judges

What you can do

• Browse system judges (vendored, ready-to-use)
• Browse your org's custom judges
• Create a new custom judge
• Test a judge against sample outputs
• GEPA-optimize a judge against labeled examples
• Apply a judge to evaluations and trace evaluations

System judges

LayerLens ships a small library of system judges for common dimensions:

• Helpfulness
• Faithfulness (RAG-shaped)
• Safety
• Tone-appropriateness
• Brevity / verbosity
• Structured-output validity

Use these as starting points; clone and customize for your team's bar.

Building a custom judge

The judge builder has 5 fields:

1. Name and description
2. Output type — binary, scored (1-5), labeled (multi-class)
3. Judging model — the LLM that runs the rubric
4. Rubric — the prompt
5. Test examples — paste samples to validate

A good rubric:

• Names the dimension explicitly
• Shows examples of "good" with labels
• Shows examples of "bad" with labels
• Specifies the output format

GEPA judge optimization

Once you have ≥30 labeled examples, run GEPA optimization. Stratix tunes the rubric prompt to maximize agreement with your labels. See Judge Optimization (GEPA).

Applying judges

Judges are reusable across:

• Evaluations (private and shared)
• Trace evaluations
• Agentic evaluations

In any of those flows, the scoring step lets you stack any number of judges alongside scorers.

Where to next

• Judge Optimization (GEPA)
• Concept: Judges
• First judge
• Tutorial: Build your first judge