Scorers

LLM-backed scorers in Stratix — model + prompt, when to use them, when to use a judge instead.

A scorer in Stratix is an LLM-backed grader — a model plus an evaluation prompt — applied to an evaluation row. Scorers and judges share the LLM-evaluation surface; the two are separated by lifecycle and usage, not by implementation kind.

	Scorer	Judge
Implementation	LLM (model + prompt)	LLM (model + rubric)
Versioning	Immutable	Versioned with execution history
Where it runs	Inside an evaluation run (benchmark or custom dataset)	Standalone, against traces or evaluation runs
Optimization	n/a	GEPA-tunable against labeled examples
Common use	Reusable rubric you apply across many evaluations	Subjective dimension you tune over time

Both call an LLM with a prompt and return a score; the distinction is "where in the workflow does it live."

Anatomy of a scorer

A scorer is a record with:

• Name (3–64 chars) and description (10–500 chars)
• Model — the LLM that runs the prompt
• Prompt — the evaluation instructions, including how to interpret inputs and return a score
• Optional organization_id / project_id for tenant-scoped scorers

When to author a scorer

• You have a scoring rubric you want to reuse across many benchmarks in your org.
• The rubric is stable — you don't expect to iterate on it with labeled examples.
• You want to apply it as part of a benchmark evaluation, not against traces.

When the rubric needs labeled-example tuning, version history, or trace-level evaluation, use a judge instead.

Authoring patterns

Rubric-style prompt

Rate the OUTPUT on a 1–5 scale for factual accuracy against the EXPECTED answer.
5 = every claim correct.
3 = mostly correct with one minor error.
1 = materially wrong.
Return only the integer.

EXPECTED: {{expected}}
OUTPUT: {{output}}

Pass/fail prompt

Does OUTPUT contain a citation that exists in CONTEXT?
Return "yes" or "no" only.

CONTEXT: {{context}}
OUTPUT: {{output}}

Structured-judgment prompt

Evaluate OUTPUT and return a JSON object:
{
 "score": <0–1>,
 "violations": [...],
 "rationale": "..."
}

INPUT: {{input}}
OUTPUT: {{output}}

Scorer composition in evaluations

An evaluation can stack multiple scorers. Each produces a per-row verdict; the evaluation's overall score is configurable:

• All-pass — every scorer must pass
• Any-pass — at least one scorer must pass
• Mean — average score across scorers
• Custom weighting — weighted aggregation

Org-scoped library

Org-scoped scorers are reusable across every benchmark in your workspace. System scorers shipped by LayerLens are available across all orgs.

Deterministic / code-based graders

Some evaluation needs are inherently deterministic — Flesch-Kincaid grade level, JSON-schema validity, regex match, statistical fairness ratios, citation-existence database lookups. These do not fit the Scorer model (which is LLM-prompt-driven). Treat them as separate code graders, computed by the evaluation runtime independently from scorers. See Custom code grader for the pattern.

Where to next

• Judges — for tunable, versioned, trace-level evaluation
• Stratix Premium — Scorers
• Evaluations
• SDK reference — Scorers