Agentic evaluation

Pre- and post-deployment quality gates for multi-step agents — combine assertions, deterministic rules, and LLM judges.

Agents fail differently from chat. They reach a wrong final state, take a wrong path to a right state, call a tool they shouldn't, or quietly regress on an edge case. Agentic evaluation is the pre- and post-deployment practice of catching these failures before they ship.

The shape of the work

1. Capture a representative trace set. Your agent's actual runs — inputs, tool calls, outputs, every span. Real or synthetic.
2. Define evaluation criteria. Mix three types:

• Natural-language assertions — "the agent identified the customer's account tier"
• Deterministic rules — "the agent never called admin_api.delete_*"
• LLM judges — "rate the helpfulness of the final response 1-5"

1. Run the evaluation. Stratix runs all criteria over the trace set in one job.
2. Read verdict + root-cause. Each failed criterion ties to the trace, the span, and the decision that broke it.
3. Detect regressions. Compare to a baseline; surface newly failing criteria.

Why it works on Stratix

• One engine, three criteria types — assertions, rules, and judges in the same evaluation
• First-class trace and span access — rules can inspect any field of any span
• Judge engine + GEPA optimization — your subjective bar gets sharper over time
• Regression detection — built-in baseline comparison
• Pre- and post-deployment fit — runs on your candidate change, not on live traffic

Tools you'll use

• Stratix Premium — Agent Evaluation
• Stratix Premium — Traces
• Stratix Premium — Judges
• SDK: client.trace_evaluations.create()

Outcomes you should see

You'll know this is working when:

• Zero CRITICAL deterministic-rule violations in any release-gate run.
• >95% pass rate on your assertion criteria across the curated trace set.
• Regression report names <2 newly failing criteria per release.
• Auditor questions about agent safety are answered live in-meeting with the verdict + root-cause artifacts.

Anti-patterns

• Judge-only evaluations. Judges are the slowest, most expensive criterion. Anchor with deterministic rules and assertions; use judges for the residual subjective bar.
• No regression detection. Without a baseline, today's pass rate is meaningless.
• Evaluating live traffic for pre-deploy gates. Live traffic is for continuous evaluation. Pre-deploy uses a captured trace set.

Where to next

• Concept: Agentic evaluation
• Overview: Agentic evaluations
• Workflow: Evaluate
• Cookbook: agentic recipes