Recipe: Score multilingual outputs

Recipe — score outputs in multiple languages with an LLM judge.

When to use: product supports multiple languages; deterministic scoring is hard across them.

Steps

1. Build judge with explicit "respond in same language as input" requirement.
2. Use a multilingual-strong judging model.
3. GEPA-optimize against labels in each language.

Code

judge = client.judges.create(
 name="multilingual-helpfulness",
 evaluation_goal="Rate helpfulness. The response must be in the SAME language as the input. Penalize language mismatch.",
 judging_model="anthropic/claude-opus-4-7"
)

See also

• Concept: Judges