Mental model
The Stratix mental model — catalog, engine, judges, traces, and how data moves through the platform.
Last updated
Was this helpful?
⌘Ctrlk
The Stratix mental model — catalog, engine, judges, traces, and how data moves through the platform.
Stratix has five primitives. Once you have these, every feature is composition.
A specific LLM (e.g., GPT-5.3, Claude Opus 4.6, Gemini 3.1 Pro). Stratix's catalog has 200+ public models plus your BYOK custom models.
A standardized input set + expected outputs. Public benchmarks (MMLU, HumanEval, GSM8K) live in the catalog; private datasets you upload live in your org.
A code grader that takes (input, output, expected) and returns a verdict. Fast, cheap, exact. Use for objective dimensions.
An LLM that takes (input, output) and a rubric and returns a verdict. Slow, expensive, fuzzy. Use for subjective dimensions. GEPA-optimize against labels.
A record of an AI call (or a chain of calls): inputs, outputs, every span, latencies, costs, errors. The unit of "what did my AI actually do."
A standard evaluation is (model, benchmark, scorers + judges) → results.
A trace evaluation is (trace_set, scorers + judges) → results.
An agentic evaluation is a trace evaluation where the criteria explicitly mix natural-language assertions, deterministic rules, and judges.
A comparison is N evaluations (each on the same benchmark, different models) viewed side-by-side.
A space is a saved evaluation configuration you re-run as conditions change.
Both flows produce the same shape of result row: per-input verdicts that aggregate into top-line scores.
Catalog stores models, benchmarks, public evaluations, public spaces (global, public)
Org-scoped storage stores private models, private benchmarks, private evaluations, judges, scorers, traces, spaces (one tenant per org)
Evaluation engine is shared infrastructure
ECU bills the compute consumed by the engine
Last updated
Was this helpful?
Was this helpful?
[Model] + [Benchmark/Dataset] → Evaluation engine → Scorers + Judges → Results
↓
(also outputs traces)
[Trace ingest] → [Trace store] → Trace evaluation engine → Scorers + Judges → Results