> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-6/bootstrap-judges.md).

# Bootstrap a judge before GEPA

Bootstrap a judge before GEPA — what to ship in week one when you don't yet have labeled data.

GEPA optimization requires labeled examples. The minimum bar varies by judge shape:

| Judge output type   | Minimum labeled examples for GEPA    |
| ------------------- | ------------------------------------ |
| Binary (pass/fail)  | ≥30                                  |
| Scored (e.g., 1-5)  | ≥50                                  |
| Labeled multi-class | ≥50 (more if classes are imbalanced) |

In week one you usually have zero. This guide shows how to ship a usable judge anyway, and how to grow into GEPA over the next 2-4 weeks.

## Week 1 — system judge + clear rubric

1. **Start from a system judge.** Stratix ships starter judges for helpfulness, faithfulness, safety, tone, brevity, and structured-output validity. Clone one whose intent is closest to your dimension.
2. **Write the rubric for your team's bar.** Plain English. Explicit examples of "good" and "bad." The rubric is what you'll later optimize; even untuned, a precise rubric outperforms a vague one.
3. **Pick a balanced judging model.** Reserve the strongest model for genuinely subtle dimensions; most rubrics work fine on a balanced choice.
4. **Test on 5-10 examples by hand.** Run the judge, read the verdicts, decide whether the rubric needs sharpening. Iterate.

This is enough to wire the judge into evaluations and trace evaluations. Treat verdicts as **directional** — a 60-70% agreement rate with humans is normal at this stage.

## Weeks 2-4 — collect labels

Every time a reviewer disagrees with the judge, capture the example as a label. Sources:

* **Triage queue.** Production traces flagged for human review. Reviewer's verdict + the AI output → labeled example.
* **CI false-positives and false-negatives.** PRs where the judge fired or didn't, and the developer disagreed.
* **Targeted labeling sprints.** Pull 30-50 representative examples from production traffic; label them in a 2-hour session.

Aim for ≥30 labels for binary judges, ≥50 for scored or multi-class.

## When you cross the threshold — run GEPA

```python
opt = client.judge_optimizations.create(
 judge_id=judge.id,
 labeled_examples_dataset_id=dataset.id,
 iterations=20,
)
result = client.judge_optimizations.wait_for_completion(opt.id)
print(f"Before: {result.before:.3f} → After: {result.after:.3f}")
```

Hold out 20% as a validation set. If validation agreement drops sharply versus training agreement, you're over-fit — expand the training set or check for label inconsistency.

## When the judge will never stabilize

Some dimensions don't converge no matter how much you tune. Signs:

* ≥3 reviewers labeling the same examples produce ≤60% inter-rater agreement
* The "good" and "bad" definitions shift between sprints
* The judge is being asked to make a judgment your team itself doesn't make consistently

In those cases, the judge isn't the problem — the dimension is under-defined. Hold a calibration session with reviewers, converge on a sharper rubric, then return to GEPA.

## See also

* [Concept: Judges](/8.-evaluate-score-the-outputs/judges-1.md) — how judges work, including the GEPA algorithm sketch
* [Tutorial 5: GEPA-optimize a judge](/9.-improve-tune-the-system/05-gepa-optimize.md)
* [Premium UI: Judge optimization](/9.-improve-tune-the-system/judge-optimization.md)
* [Standardize judges across teams](/more-in-this-section-6/standardize-judges.md)