Agentic evaluations — overview

Pre- and post-deployment quality gates for multi-step AI agents — assertions, deterministic rules, and LLM judges.

Most AI features today are not single-turn chat. They're agents — multi-step systems that plan, call tools, retrieve context, and decide when they're done. Single-output evaluation isn't enough for agents. You need to know whether the agent reached the right state, took legal steps to get there, and won't regress when the prompt changes next week.

That's what Agentic Evaluations does.

The failure scenarios

Agents fail in ways simple chat doesn't. The Stratix Agentic Evaluations product is shaped around these specific failures:

• Wrong final state. The agent answered the question but updated the wrong record.
• Right answer, wrong path. It got there, but called an internal API it shouldn't have.
• Catastrophic tool selection. A delete_user call when the user asked to "deactivate."
• Off-policy reasoning. The agent's chain-of-thought reveals reasoning your team doesn't want shipped (PII, leaked instructions, harmful claims).
• Silent regression. A prompt change five PRs ago broke an edge case nobody is watching.

Three evaluation criteria types, combined

Agentic evaluations let you mix three criteria types in one evaluation:

1. Natural-language assertions. Plain-English checks the LLM grades. "The agent correctly identified the customer's account tier." Cheap, flexible, good for fuzzy correctness.
2. Deterministic rules. Code or schema checks. "The agent never called admin_api.* outside the allowed scope." Fast, cheap, exact.
3. LLM judges. Subjective dimensions evaluated by an optimized judge. "How helpful was the final response on a 1-5 scale?" Use sparingly — most subjective bars resolve into combinations of assertions.

A typical agentic evaluation uses all three. Deterministic rules catch hard violations early. Assertions cover the bulk of correctness. Judges grade the residual subjective quality.

The pre- and post-deployment workflow

The Stratix Agentic Evaluations workflow is built for pre- and post-deployment — you run it on a candidate change before it ships, not on live production.

1. Capture a representative trace set. Real or synthetic, agent inputs + outputs + every span.
2. Define your evaluation criteria. A mix of assertions, rules, and judges in an evaluation space.
3. Run the evaluation. Stratix replays the criteria over the trace set.
4. Read the verdict and root-cause. Each failed criterion ties back to the trace, the span, the input, and the agent decision that broke it.
5. Detect regressions. Compare the run to a baseline; surface the criteria that newly failed.

The output artifacts are:

• A verdict (pass/fail and severity per criterion)
• A root-cause report (which span, which input, which decision)
• A regression report (what newly fails compared to baseline)

You wire those into a CI gate, a release-readiness review, or a Slack alert.

What you get vs what you build

You build:

• The trace set (your data, your agent runs)
• The evaluation criteria (your assertions, your rules, your judges)
• The CI/CD wiring

Stratix gives you:

• The evaluation engine that runs all three criteria types in one job
• The judge engine and the GEPA optimizer
• The deterministic-rule executor with first-class span access
• The verdict, root-cause, and regression reports
• The Premium UI for browsing failures and the SDK for automation

When NOT to use Agentic Evaluations

• For single-turn chat with no tools or chain, the Model Evaluations product is simpler. Pick that.
• For post-deployment monitoring of live traffic, trace evaluations is the right shape — same engine, recurring schedule, applied to real production traces.

Where to next

• Use case: Agentic evaluation
• Concept: Agentic evaluation
• Workflow: Evaluate
• Stratix Premium: Agent Evaluation