> For the complete documentation index, see [llms.txt](https://docs.layerlens.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.layerlens.ai/more-in-this-section-6/bootstrap-judges.md).

# Bootstrap a judge before GEPA

GEPA optimization requires labeled examples. The minimum bar varies by judge shape:

| Judge output type   | Minimum labeled examples for GEPA    |
| ------------------- | ------------------------------------ |
| Binary (pass/fail)  | ≥30                                  |
| Scored (e.g., 1-5)  | ≥50                                  |
| Labeled multi-class | ≥50 (more if classes are imbalanced) |

In week one you usually have zero. This guide shows how to ship a usable judge anyway, and how to grow into GEPA over the next 2-4 weeks.

## Week 1 — system judge + clear rubric

1. **Start from a system judge.** Stratix ships starter judges for helpfulness, faithfulness, safety, tone, brevity, and structured-output validity. Clone one whose intent is closest to your dimension.
2. **Write the rubric for your team's bar.** Plain English. Explicit examples of "good" and "bad." The rubric is what you'll later optimize; even untuned, a precise rubric outperforms a vague one.
3. **Pick a balanced judging model.** Reserve the strongest model for genuinely subtle dimensions; most rubrics work fine on a balanced choice.
4. **Test on 5-10 examples by hand.** Run the judge, read the verdicts, decide whether the rubric needs sharpening. Iterate.

This is enough to wire the judge into evaluations and trace evaluations. Treat verdicts as **directional** — a 60-70% agreement rate with humans is normal at this stage.

## Weeks 2-4 — collect labels

Every time a reviewer disagrees with the judge, capture the example as a label. Sources:

* **Triage queue.** Production traces flagged for human review. Reviewer's verdict + the AI output → labeled example.
* **CI false-positives and false-negatives.** PRs where the judge fired or didn't, and the developer disagreed.
* **Targeted labeling sprints.** Pull 30-50 representative examples from production traffic; label them in a 2-hour session.

Aim for ≥30 labels for binary judges, ≥50 for scored or multi-class.

## When you cross the threshold — run GEPA

```python
opt = client.judge_optimizations.create(
 judge_id=judge.id,
 labeled_examples_dataset_id=dataset.id,
 iterations=20,
)
result = client.judge_optimizations.wait_for_completion(opt.id)
print(f"Before: {result.before:.3f} → After: {result.after:.3f}")
```

Hold out 20% as a validation set. If validation agreement drops sharply versus training agreement, you're over-fit — expand the training set or check for label inconsistency.

## When the judge will never stabilize

Some dimensions don't converge no matter how much you tune. Signs:

* ≥3 reviewers labeling the same examples produce ≤60% inter-rater agreement
* The "good" and "bad" definitions shift between sprints
* The judge is being asked to make a judgment your team itself doesn't make consistently

In those cases, the judge isn't the problem — the dimension is under-defined. Hold a calibration session with reviewers, converge on a sharper rubric, then return to GEPA.

## See also

* [Concept: Judges](/8.-evaluate-score-the-outputs/judges-1.md) — how judges work, including the GEPA algorithm sketch
* [Tutorial 5: GEPA-optimize a judge](/9.-improve-tune-the-system/05-gepa-optimize.md)
* [Premium UI: Judge optimization](/9.-improve-tune-the-system/judge-optimization.md)
* [Standardize judges across teams](/more-in-this-section-6/standardize-judges.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.layerlens.ai/more-in-this-section-6/bootstrap-judges.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.