Detect and remove flaky scorers

Guide — detect and remove flaky scorers.

Flaky scorers cause CI to fail on noise. Detect by:

1. Re-run the same evaluation 3 times back-to-back
2. Compute scorer-by-scorer variance
3. Scorers with high variance on identical inputs are flaky

Fix: pin model temperature to 0; add deterministic post-processing; or replace with a code grader if possible.

See also

• Concept: Scorers