Evaluations
An evaluation in Stratix — what it actually is, what it produces, how it's stored.
Last updated
Was this helpful?
⌘Ctrlk
An evaluation in Stratix — what it actually is, what it produces, how it's stored.
An evaluation in Stratix is a discrete run that produces results. It's the unit of measurement.
Every evaluation has:
Model(s) — what's being evaluated
Dataset / benchmark — the inputs (and, for benchmarks, the expected outputs)
Scoring config — the scorers and/or judges
Configuration metadata — a name, description, tags, owner
A run history — every time you (re-)run, a new result is captured
Public evaluations — visible on stratix.layerlens.ai, citable, contributed by LayerLens or partners (2,000+)
Private evaluations — scoped to your org in stratix.layerlens.ai
Both flow through the same evaluation engine. The only difference is visibility and tenant scoping.
A result has:
Top-line score(s) — one per scoring dimension
Per-row results — for each input row: input, output, expected (if any), per-scorer/judge verdicts
Cost and latency — per row and rolled up
Status — running, completed, failed, partial
Run metadata — model used, scoring config used, timestamp, owner
Four shapes of evaluation:
One model, one dataset, scoring config. Most common.
Two or more models, one dataset, scoring config. Side-by-side scores.
A trace set as the input, scoring config applied. See Traces and spans.
A trace evaluation where the criteria explicitly mix natural-language assertions, deterministic rules, and LLM judges. See Agentic evaluation.
All four are stored as evaluations in the same surface and share the same result shape.
Re-running an evaluation creates a new run record under the same evaluation. Score-over-time charts use the run history.
Last updated
Was this helpful?
Was this helpful?