Evaluations

client.evaluations — create benchmark and dataset evaluations, wait for completion, list, retrieve results.

Create

The canonical pattern is to fetch model and benchmark objects first, then pass them to create (not just IDs):

from layerlens import Stratix

client = Stratix()

# Fetch the objects you want to evaluate
model = client.models.get_by_key("openai/gpt-4o")
benchmark = client.benchmarks.get_by_key("arc-agi-2")

# Create the evaluation
evaluation = client.evaluations.create(
 model=model,
 benchmark=benchmark,
)
print(f"Created evaluation {evaluation.id}, status={evaluation.status}")

get_by_key is case-sensitive. Alternatively, fetch a list and pick:

models = client.models.get(type="public", name="gpt-4o")
model = models[0]

Wait for completion

Pass the evaluation object (not just the ID) to wait_for_completion:

evaluation = client.evaluations.wait_for_completion(
 evaluation,
 interval_seconds=10,
 timeout_seconds=600, # 10 minutes
)
print(f"Evaluation {evaluation.id} finished with status={evaluation.status}")

Once the evaluation completes, retrieve results:

if evaluation.is_success:
 results = client.results.get(evaluation=evaluation)
 print("Results:", results)
else:
 print(f"Evaluation did not succeed: {evaluation.status}")

The Evaluation object exposes .is_success, .status, .id, and other fields directly.

List and filter

# Get many evaluations
response = client.evaluations.get_many()

# Filter by status, accuracy, date — see samples/core/evaluation_filtering.py
filtered = client.evaluations.get_many(
 status="completed",
 sort_by="created_at",
 sort_order="desc",
 page_size=20,
)

Async

Every method has an awaitable counterpart on AsyncStratix:

import asyncio
from layerlens import AsyncStratix

async def main():
 client = AsyncStratix()
 model = await client.models.get_by_key("openai/gpt-4o")
 benchmark = await client.benchmarks.get_by_key("arc-agi-2")
 evaluation = await client.evaluations.create(model=model, benchmark=benchmark)
 result = await client.evaluations.wait_for_completion(evaluation)
 print(f"Accuracy: {result.accuracy}")

asyncio.run(main())

For concurrent runs, see samples/core/async_workflow.py.

Result fields

The result object exposes:

accuracy — overall accuracy for benchmark-style evaluations
status — "queued", "running", "completed", "failed", "cancelled"
is_success — boolean shortcut for status == "completed"
id — stable evaluation ID

For per-row data and detailed scores, retrieve via client.results.get(evaluation=evaluation).

Compare

# Compare results across multiple evaluation runs — see samples/core/compare_evaluations.py

Source samples

Sample

What it shows

samples/core/quickstart.py

Minimal end-to-end

samples/core/run_evaluation.py

Full evaluation lifecycle

samples/core/benchmark_evaluation.py

Model vs. benchmark

samples/core/async_workflow.py

Concurrent evaluations

samples/core/evaluation_filtering.py

Filter and paginate

samples/core/compare_evaluations.py

Compare runs

samples/core/evaluation_pipeline.py

Chain judges + traces + results

Good evening