Tutorial 1: First evaluation in 10 minutes

Tutorial 1 — Run your first end-to-end Stratix evaluation in 10 minutes.

Time: ~10 minutes Level: Beginner You'll build: A complete evaluation against a public benchmark, programmatically and via the UI.

What you'll learn

• How to run a Stratix evaluation from the SDK
• How to read evaluation results
• How to compare two models on the same benchmark

Prerequisites

• Stratix Premium account (sign up)
• Python 3.9+
• pip install layerlens --extra-index-url https://sdk.layerlens.ai/package
• LAYERLENS_STRATIX_API_KEY exported

Step 1: Verify the SDK

python -c "import layerlens; print(layerlens.__version__)"

Checkpoint: 1.3.0 (or later).

Step 2: Pick a model and a benchmark

from layerlens import Stratix

client = Stratix()

model = client.models.get_by_key("openai/gpt-4o")
benchmark = client.benchmarks.get_by_key("arc-agi-2")

print(model.name, "→", benchmark.name)

Step 3: Create the evaluation

evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
)
print("Started:", evaluation.id)

Step 4: Wait for completion

result = client.evaluations.wait_for_completion(evaluation)
print(f"Accuracy: {result.accuracy}")

Checkpoint: You should see an accuracy score.

Step 5: Inspect in the UI

The same evaluation appears in Premium → Evaluations. Click any failing row to see input, output, expected, verdict.

Step 6: Compare two models

model_a = client.models.get_by_key("openai/gpt-4o")
model_b = client.models.get_by_key("anthropic/claude-opus-4-7")

eval_a = client.evaluations.create(model=model_a, benchmark=benchmark)
eval_b = client.evaluations.create(model=model_b, benchmark=benchmark)

result_a = client.evaluations.wait_for_completion(eval_a)
result_b = client.evaluations.wait_for_completion(eval_b)

print(f"GPT-4o: {result_a.accuracy}")
print(f"Claude Opus 4: {result_b.accuracy}")

What's next

• Tutorial 2: First judge — score subjective dimensions
• Cookbook — pick a recipe matching your task
• Concept: Evaluations