Bootstrap a judge before GEPA
Bootstrap a judge before GEPA — what to ship in week one when you don't yet have labeled data.
Last updated
Was this helpful?
⌘Ctrlk
Bootstrap a judge before GEPA — what to ship in week one when you don't yet have labeled data.
GEPA optimization requires labeled examples. The minimum bar varies by judge shape:
Binary (pass/fail)
≥30
Scored (e.g., 1-5)
≥50
Labeled multi-class
≥50 (more if classes are imbalanced)
In week one you usually have zero. This guide shows how to ship a usable judge anyway, and how to grow into GEPA over the next 2-4 weeks.
Start from a system judge. Stratix ships starter judges for helpfulness, faithfulness, safety, tone, brevity, and structured-output validity. Clone one whose intent is closest to your dimension.
Write the rubric for your team's bar. Plain English. Explicit examples of "good" and "bad." The rubric is what you'll later optimize; even untuned, a precise rubric outperforms a vague one.
Pick a balanced judging model. Reserve the strongest model for genuinely subtle dimensions; most rubrics work fine on a balanced choice.
Test on 5-10 examples by hand. Run the judge, read the verdicts, decide whether the rubric needs sharpening. Iterate.
This is enough to wire the judge into evaluations and trace evaluations. Treat verdicts as directional — a 60-70% agreement rate with humans is normal at this stage.
Every time a reviewer disagrees with the judge, capture the example as a label. Sources:
Triage queue. Production traces flagged for human review. Reviewer's verdict + the AI output → labeled example.
CI false-positives and false-negatives. PRs where the judge fired or didn't, and the developer disagreed.
Targeted labeling sprints. Pull 30-50 representative examples from production traffic; label them in a 2-hour session.
Aim for ≥30 labels for binary judges, ≥50 for scored or multi-class.
Hold out 20% as a validation set. If validation agreement drops sharply versus training agreement, you're over-fit — expand the training set or check for label inconsistency.
Some dimensions don't converge no matter how much you tune. Signs:
≥3 reviewers labeling the same examples produce ≤60% inter-rater agreement
The "good" and "bad" definitions shift between sprints
The judge is being asked to make a judgment your team itself doesn't make consistently
In those cases, the judge isn't the problem — the dimension is under-defined. Hold a calibration session with reviewers, converge on a sharper rubric, then return to GEPA.
Concept: Judges — how judges work, including the GEPA algorithm sketch
Last updated
Was this helpful?
Was this helpful?
opt = client.judge_optimizations.create(
judge_id=judge.id,
labeled_examples_dataset_id=dataset.id,
iterations=20,
)
result = client.judge_optimizations.wait_for_completion(opt.id)
print(f"Before: {result.before:.3f} → After: {result.after:.3f}")