Scorers
The Stratix Premium Scorers surface — manage LLM-backed scorers org-scoped.
Last updated
Was this helpful?
⌘Ctrlk
The Stratix Premium Scorers surface — manage LLM-backed scorers org-scoped.
Available in Stratix Premium. Scorers are a logged-in workspace feature at stratix.layerlens.ai. Stratix Public users can browse the catalog but not author or run scorers.
Scorers are LLM-backed graders — a model plus an evaluation prompt — that you apply across benchmarks. Your org builds a library of them over time.
A scorer record contains:
Name (3–64 chars) and description (10–500 chars)
Model — the LLM that runs the prompt (pick from the catalog or a BYOK model)
Prompt — the evaluation instructions
Scorers are immutable once created. To revise a rubric, author a new scorer; this keeps prior evaluation results reproducible against the exact rubric that produced them.
Scorer — reusable rubric applied as part of a benchmark or custom-dataset evaluation. Immutable. No versioning, no trace-level surface.
Judge — versioned rubric tuned with labeled examples, applied to traces (and optionally to evaluation runs). Use a judge when you need GEPA tuning or trace-level evaluation.
Click New scorer:
Name and description — the catalog labels.
Pick a model — frontier for nuanced rubrics; cost-optimized for high-volume, narrow rubrics.
Author the prompt — include input-variable placeholders ({{output}}, {{expected}}, {{context}}, {{input}} as relevant) and tell the model exactly what to return (numeric range, label set, structured JSON).
Test on examples — paste sample inputs/outputs, verify the scorer returns expected verdicts.
Save — scorer becomes available in any evaluation in your org.
When you create an evaluation, the scoring step lets you stack any number of scorers. Each scorer produces a per-row verdict; the evaluation's overall score is configurable (mean, median, all-pass, etc.).
Some checks are inherently deterministic — exact match, regex, JSON-schema validity, Flesch-Kincaid grade, fairness math. These don't fit the LLM-prompt Scorer surface and run as separate code graders in the evaluation runtime. See Custom code grader recipe.
Last updated
Was this helpful?
Was this helpful?