Your first judge
Build your first LLM judge in Stratix Premium — define a rubric, test it, optionally GEPA-optimize.
A judge is an LLM that scores subjective dimensions — helpfulness, faithfulness, tone, safety — that a code grader can't grade. Building one takes about 15 minutes.
Prerequisites
Steps
1. Open Judges
Left rail: Agent Evaluation → Judges. Click New judge.
2. Name and describe
Name: e.g., "helpfulness-customer-support"
Description: what this judge grades, in one sentence
Output type: binary (pass/fail), score (e.g., 1-5), or labeled (e.g., "helpful" / "neutral" / "unhelpful")
3. Pick a judging model
The model that runs the rubric. Default is a balanced choice; pick a stronger model for harder rubrics.
4. Write the rubric
The rubric is the prompt the judging model uses. Best practices:
State the dimension you're grading
Describe what "good" looks like with examples
Describe what "bad" looks like with examples
Show the output format you want back
The Premium UI provides a starter template you can adapt.
5. Test on a few examples
Paste 3-5 sample outputs and run the judge. Read the verdicts. If they don't match your intuition, iterate the rubric.
6. Optional: GEPA-optimize
If you have ≥30 labeled examples, run GEPA optimization. Stratix tunes the rubric to better match your labels. Agreement with humans typically rises 10-20 percentage points.
7. Save
The judge is now reusable in any evaluation, trace evaluation, or agentic evaluation in your org.
Verify
You should be able to apply your judge to a single sample input/output and get back a verdict.
What to try next
Apply the judge to your first evaluation. Open the eval, add the judge, rerun.
Apply the judge to a trace evaluation. Real production data, judged.
Build a second judge for a different dimension. Stratix evaluations let you stack many judges.
Where to next
Last updated
Was this helpful?