Agentic evaluation
Agentic evaluation — pre- and post-deployment quality gates for multi-step AI agents.
Last updated
Was this helpful?
⌘Ctrlk
Agentic evaluation — pre- and post-deployment quality gates for multi-step AI agents.
Agentic evaluation is the pre- and post-deployment practice of grading multi-step agents. Three criteria types combined into one evaluation: assertions, deterministic rules, and judges.
Single-output evaluation (input → output → score) misses the path. Agents fail in ways simple chat doesn't:
They reach a wrong final state
They take a wrong path to a right state
They call a tool that should have been off-limits
Their chain-of-thought reveals reasoning your team doesn't want shipped
They quietly regress on an edge case
You need to grade the whole trace — every span, every decision — not just the final output.
Plain-English checks the LLM grades. "The agent correctly identified the customer's account tier." Cheap, flexible, good for fuzzy correctness.
Code or schema checks. "The agent never called admin_api.delete_*." Fast, cheap, exact.
Subjective dimensions evaluated by an optimized judge. "How helpful was the final response on a 1-5 scale?" Use sparingly; anchor with assertions and rules.
70% deterministic rules — hard correctness, hard policy violations
20% natural-language assertions — fuzzy correctness
10% LLM judges — residual subjective bar
This shape keeps evaluations fast and predictable.
Agentic evaluations target candidate changes: a new prompt, a new model, a new tool. They run on a captured trace set, not on live traffic. (For live traffic, see Continuous evaluation.)
Verdict — pass/fail and severity per criterion
Root-cause report — which trace, which span, which decision broke
Regression report — what newly fails compared to baseline
CI gates on agent code/prompt changes
Release-readiness reviews
Auditor-friendly evidence in regulated industries
Last updated
Was this helpful?
Was this helpful?