Tutorial 4: Score live traces

Tutorial 4 — Score production traces continuously with Stratix trace evaluations.

Time: ~30 minutes Level: Intermediate You'll build: A continuous trace evaluation running on real production traffic.

What you'll learn

• How to ingest traces from your application
• How to build a trace evaluation
• How to schedule continuous runs
• How to alert on score drift

Prerequisites

• Completed Tutorials 1 and 2
• An application emitting AI calls (real or simulated)
• A scoring config (scorers + judges) you want to apply

Step 1: Ingest traces

In your application, after each AI call:

from layerlens import Stratix
client = Stratix()

client.traces.create(
 id=correlation_id,
 name="answer-question",
 inputs={"prompt": prompt_text},
 outputs={"response": response_text},
 spans=[
 {"name": "retrieval", "kind": "tool", "inputs": {...}, "outputs": {...}},
 {"name": "llm", "kind": "llm", "model": "claude-opus-4-7", "inputs": {...}, "outputs": {...}}
 ],
 tags={"env": "production", "feature": "support-bot"}
)

For batch ingestion:

client.traces.upload("./traces.jsonl")

Step 2: Build the trace evaluation

trace_eval = client.trace_evaluations.create(
 trace_set={"tags": {"env": "production", "feature": "support-bot"}},
 scorers=["scorer-id-1"],
 judges=["judge-id-1"],
 schedule="hourly",
)
print(trace_eval.id)

Step 3: Watch a few runs

After an hour, scores start populating. Open Premium → Agent Evaluation → Trace evaluations → your trace eval. The score-over-time chart fills in.

Step 4: Configure thresholds

client.trace_evaluations.set_alert(
 trace_eval_id=trace_eval.id,
 metric="judge.helpfulness.pass_rate",
 threshold=0.85,
 direction="below",
)

When pass rate drops below 85%, you get notified.

Step 5: Watch the trend

Over a week, the score-over-time chart tells you whether quality is steady or drifting. Drift is the early-warning signal you wanted.

What's next

• Tutorial 5: GEPA-optimize a judge
• Concept: Continuous evaluation
• Workflow: Observe
• Stratix Premium — Trace evaluations