Recipe: compare prompts on a small test set

Recipe — A/B compare prompt variants. Pointer to the canonical SDK sample.

The canonical sample lives in the SDK: samples/core/compare_evaluations.py — compare evaluation runs side-by-side to measure improvement over time.

Pattern

1. Hold model and dataset fixed; vary the prompt.

2. Create an evaluation per variant with client.evaluations.create(model=, benchmark=).

3. Wait for completion on each via client.evaluations.wait_for_completion(evaluation).

4. Compare top-line accuracy and per-result deltas.

See also

• SDK sample: compare_evaluations.py

• Tutorial 1: First evaluation

• Workflow: Improve