Scorers

client.scorers — LLM-backed scorers (model + prompt). CRUD reference.

In Stratix, custom scorers are LLM-backed: a scorer wraps a model + prompt. Use scorers when you need a custom scoring rule that doesn't fit the built-in benchmark metrics or judge surfaces.

Create

scorer = client.scorers.create(
 name="response-rubric",
 description="Rate response on factual accuracy",
 model_id=model.id,
 prompt="Rate the response on a 1-5 scale based on factual accuracy...",
)

Get

scorer = client.scorers.get(scorer_id)
# Returns Scorer with: id, organization_id, project_id, name, description,
# model_id, model_name, model_key, model_company, prompt, created_at, updated_at

List

response = client.scorers.get_many(page=1, page_size=20)

Update

client.scorers.update(
 scorer_id,
 name="...", # any subset of fields
 description="...",
 model_id="...",
 prompt="...",
)

Delete

client.scorers.delete(scorer_id)

Where scorers fit

• Custom benchmark evaluation — pass custom_scorer_ids=[...] to client.benchmarks.create_custom(...) to apply scorers to that benchmark.
• Trace evaluation — judges are typically the path; scorers are used when the rubric is reusable across many benchmarks.

Async

All methods have AsyncStratix equivalents.

See also

• Concept: Scorers
• Stratix Premium — Scorers
• Judges — LLM-as-a-judge for trace evaluations