Recipe: Score reasoning chains

Recipe — score reasoning chains with chain-of-thought aware judges.

When to use: model exposes its reasoning trace; you need the trace to be valid not just the final answer.

Steps

1. Capture chain-of-thought as a span (kind: chain).
2. Build judge that scores both final answer + reasoning soundness.
3. Penalize reasoning errors that happen to land on a correct answer.

Code

judge = client.judges.create(
 name="reasoning-soundness",
 evaluation_goal="Rate (1) final-answer correctness and (2) reasoning soundness. Penalize correct answers that arrived via flawed reasoning."
)

See also

• Concept: Traces and spans
• Concept: Judges