Recipe: create a custom scorer

Recipe — create a custom scorer (LLM-backed). Pointer to the SDK scorer + judge-creation samples.

In Stratix, custom scorers are LLM-backed: a scorer wraps a model + a prompt. The SDK shape:

scorer = client.scorers.create(
 name="response-rubric",
 description="Rate response on factual accuracy",
 model_id=model.id,
 prompt="Rate the response on a 1-5 scale...",
)

CRUD covers create, get, get_many(page=, page_size=), update(id,...), delete(id). Returned Scorer includes model_id, model_name, model_company, prompt.

For a complete custom-judge-with-validation example, see samples/core/judge_creation_and_test.py — builds a custom PII detection judge and validates against known inputs.

For deterministic logic (regex, JSON-schema, exact match), encode the rule in the scorer's prompt or use a judge with rule-based goals.

See also

• SDK reference: scorers
• Concept: Scorers
• Stratix Premium — Scorers