Continuous evaluation

Continuous evaluation — score live production traces on a recurring schedule.

Continuous evaluation scores live production traces on a recurring schedule. Pre- and post-deployment evaluations catch regressions before merge; continuous evaluation catches them after.

Why it matters

• Production traffic shifts; what worked in pre-deploy may fail in the wild
• Model providers update their backends; your scores may move quietly
• Prompt or feature changes that slipped through CI surface here

Without continuous evaluation, AI quality is a post-incident discovery. With it, regressions surface before customer-facing impact.

The shape

1. Ingest live traces — from your application, via SDK or API
2. Define a trace evaluation — scorers + judges
3. Schedule it — daily, hourly, per-batch
4. Watch the trend — score-over-time per dimension
5. Get notified on drift — thresholds, alerts

Pre-deploy vs continuous patterns

Aspect	Pre-deploy agentic	Continuous
Trace set	curated, fixed	rolling, sampled production
Cadence	per-PR, per-release	daily, hourly
Bar	block merge	alert on drift
Cost profile	bursty	steady

Sampling

For high-volume traffic, sample. Sample bias is the trap — production distribution drifts, and sampling drifts with it. Re-validate the sampling strategy periodically.

Cost management

Continuous evaluation can become a big consumer of ECU. Two levers:

• Sample less of low-importance traffic, more of high-stakes
• Use scorers liberally; reserve judges for residual subjective bar

Where to next

• Trace evaluations (Premium)
• Stratix Premium — Notifications
• Use case: Continuous evaluation
• Tutorial: Score live traces