Run your first evaluation in 5 minutes

Run your first Stratix evaluation end-to-end in five minutes. One Python file, copy-paste-runnable.

One Python file. One model. One benchmark. One score. That's it.

This page is the fastest possible path from zero to a real evaluation. If something breaks, jump straight to Common first-day errors and come back.

Prereqs: Python 3.9+, a Stratix Premium account, and an API key from Settings → API keys. You'll need ~10 ECU of starter credit (every new org gets some).

Step 1 — Install and authenticate (60 seconds)

pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
export LAYERLENS_STRATIX_API_KEY="sk-..." # macOS / Linux
# PowerShell: $env:LAYERLENS_STRATIX_API_KEY = "sk-..."

Verify the install:

python -c "import layerlens; print(layerlens.__version__)"

You should see 1.3.0 or later. If not, see first-day errors.

Step 2 — The whole thing in one file

Save this as first_eval.py and run it. That's the entire 5-minute experience.

"""
first_eval.py — your first end-to-end Stratix evaluation.

Run with: python first_eval.py
Requires: LAYERLENS_STRATIX_API_KEY set in your environment.
"""
from layerlens import Stratix

# 1) Authenticate. The Stratix() constructor reads LAYERLENS_STRATIX_API_KEY
# from the environment. Pass api_key="..." explicitly if you prefer.
client = Stratix()

# 2) Pick a model from the public catalog. get_by_key() is case-sensitive
# and uses the "<provider>/<model>" key shown in the Catalog UI.
model = client.models.get_by_key("anthropic/claude-opus-4-7")
print(f"Model: {model.name}")

# 3) Pick a benchmark. MMLU is a fast, well-known public benchmark — good
# for a smoke test on day one.
benchmark = client.benchmarks.get_by_key("mmlu")
print(f"Benchmark: {benchmark.name}")

# 4) Create the evaluation. The platform queues it; the SDK returns
# immediately with an Evaluation object you can poll.
evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
)
print(f"Started: {evaluation.id} (status={evaluation.status})")

# 5) Block until the evaluation finishes. wait_for_completion polls every
# interval_seconds and gives up after timeout_seconds. MMLU on Opus 4.7
# typically completes in 2–4 minutes.
evaluation = client.evaluations.wait_for_completion(
 evaluation,
 interval_seconds=10,
 timeout_seconds=600,
)

if not evaluation.is_success:
 raise SystemExit(f"Evaluation did not succeed: status={evaluation.status}")

print(f"Done: accuracy={evaluation.accuracy:.3f}")

# 6) Pull per-prompt results. get_all_by_id auto-paginates across pages and
# returns a flat list of Result objects (.prompt,.result,.truth,.score).
results = client.results.get_all_by_id(evaluation.id)
print(f"Rows: {len(results)}")

# 7) Show the first three rows so you can see what real data looks like.
for r in results[:3]:
 verdict = "PASS" if r.score and r.score >= 0.5 else "FAIL"
 print(f" [{verdict}] {r.prompt[:60]}... score={r.score}")

# 8) Done. The same evaluation now appears in Premium → Evaluations.
print(f"\nView in UI: https://stratix.layerlens.ai/evaluations/{evaluation.id}")

Step 3 — Run it

python first_eval.py

You should see something like:

Model: Claude Opus 4.7
Benchmark: MMLU
Started: eval_01HZ... (status=queued)
Done: accuracy=0.873
Rows: 14042
 [PASS] Which of the following best describes the function of... score=1.0
 [PASS] In the context of macroeconomic policy, the term 'cro... score=1.0
 [FAIL] A 65-year-old patient presents with acute onset chest... score=0.0

View in UI: https://stratix.layerlens.ai/evaluations/eval_01HZ...

Open that URL. The dashboard shows the same accuracy number, plus a per-subset breakdown, error analysis, and ECU consumed.

You just did five things

1. Selected a model and a benchmark from the public catalog.
2. Built the evaluation by passing both objects to evaluations.create.
3. Observed the run via wait_for_completion (queued → running → completed).
4. Evaluated the model against MMLU's ground-truth labels — an LLM-backed scorer compares each model response to truth and emits a 0/1 score per row.
5. Improved your understanding of where the model wins and loses, by inspecting per-row results.

Those five stages — Select, Build, Observe, Evaluate, Improve — are the spine of every Stratix workflow.

Common failure modes

• AuthenticationError: invalid API key — env var name is LAYERLENS_STRATIX_API_KEY, not LAYERLENS_API_KEY. See first-day errors.
• NotFoundError on get_by_key — keys are case-sensitive. Browse the catalog at https://stratix.layerlens.ai/catalog to copy the exact key.
• Timeout — bump timeout_seconds. Some benchmarks (HumanEval+, GPQA) take longer than 10 minutes on slower models.
• "No ECU" — every new org gets starter ECU; if you've exhausted it, top up under Admin → Billing.

Where to next

You ran one model on one benchmark. Logical next steps:

• Compare two models. Repeat Steps 2–5 with a different model key and diff the accuracy. Public — compare two models walks the UI version of this flow.
• Score subjective dimensions. Accuracy works when there's a ground-truth answer. For helpfulness, faithfulness, tone, etc., you need an LLM judge. Tutorial 2: First judge.
• Run it on your own data. Swap the benchmark for an uploaded dataset. Upload a private benchmark.
• Wire this into CI. Promote your first_eval.py into a quality gate. Tutorial 3: CI/CD gates.
• Score live traces. Score traffic that's already happening in production. Tutorial 4: Score live traces.

If you'd rather follow a guided five-tutorial path, start at All tutorials. If you want to lift-and-shift a working snippet for a specific problem, jump to Top 5 starter recipes.