Tutorial 2: Build your first judge

Tutorial 2 — Build, test, and apply your first LLM judge.

Time: ~20 minutes Level: Beginner-intermediate You'll build: A custom helpfulness judge, tested against examples, applied to an evaluation.

What you'll learn

• How to author a judge rubric
• How to test a judge against examples
• How to apply a judge to an evaluation
• When to GEPA-optimize (next tutorial)

Prerequisites

• Completed Tutorial 1
• At least 10 example outputs you can label "good" or "bad"

Step 1: Define the dimension

Write down — in plain English — what you're trying to grade. Keep it specific. "Helpfulness for customer-support questions about billing." Not just "helpful."

Step 2: Create the judge via SDK

from layerlens import Stratix

client = Stratix()

judge = client.judges.create(
 name="billing-support-helpfulness",
 evaluation_goal="Rate whether the response helps the customer resolve a billing question. The response should: (1) directly address the question, (2) provide actionable next steps, (3) be polite and professional.",
)
print("Judge:", judge.id)

Step 3: Test on examples

sample_input = "I was charged twice this month. What do I do?"
sample_output = "Sorry to hear that. Please email [email protected] and include the dates of both charges. We'll refund the duplicate within 3-5 business days."

verdict = client.judges.run(
 judge_id=judge.id,
 input=sample_input,
 output=sample_output,
)
print("Verdict:", verdict.score, verdict.reasoning)

Step 4: Iterate the rubric

If the verdict doesn't match your intuition, edit the rubric and re-test.

client.judges.update(
 judge_id=judge.id,
 evaluation_goal="(updated rubric...)",
)

Step 5: Apply to an evaluation

evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
 judges=[judge.id],
)
result = client.evaluations.wait_for_completion(evaluation)
print(f"Helpfulness pass rate: {result.judge_results[judge.id].pass_rate}")

Step 6 (optional): GEPA-optimize

If you have ≥30 labeled examples, jump to Tutorial 5 to lift judge agreement.

What's next

• Tutorial 3: CI/CD quality gates
• Tutorial 5: GEPA-optimize a judge
• Concept: Judges