Agent Testing
Testing a Non-Deterministic System
Conventional tests assert that an input produces an output. An agent given the same input twice will produce two different outputs, both of which may be correct. Agent testing therefore asserts on properties rather than strings: that a disclosure was made, that a tool was called with the right identifier, that the conversation reached a resolution within a budget. The hard part is not running the tests but writing criteria that stay meaningful as the wording moves underneath them.
Four Test Shapes
They differ in how much of the conversation is real, how much is fixture, and what the assertion is allowed to touch.
The conversation history is written by hand up to a chosen point, the agent produces one further turn, and the assertion applies to that turn only. The starting state is a fixture, identical on every run.
A scripted persona converses with the agent from the opening turn, bounded by a turn limit, with tools usually mocked. The assertion is on the outcome and optionally on the path taken to reach it.
Checks which tool was selected and with what arguments, independent of the surrounding language. The only agent assertion that is fully deterministic once the call has been made.
A real conversation captured from production, converted into a fixture, and re-run against each change. The suite grows from incidents rather than imagination.
Next-Reply Against Simulation
These two look like variations of one another and behave like unit tests and integration tests. A next-reply test hand-authors the conversation history up to a chosen point, lets the agent produce exactly one more turn, and asserts on that turn alone. A simulation starts from nothing, drives a scripted persona through a whole conversation under a turn limit, and asserts on the outcome.
| Dimension | Next-Reply | Simulation |
|---|---|---|
| Starting state | Hand-written history, identical every run | Empty, reached by driving the conversation |
| What runs | Exactly one agent turn | Many turns, bounded by a limit |
| Assertion | On the single reply produced | On the outcome, and optionally the path |
| Variance | Low. Only one generation differs | High. Every turn compounds the last |
| Cost per run | One generation plus a judge | Two agents talking for the length of the limit |
| Catches | A wrong decision at a known state | Never reaching the state, loops, dead ends |
| Misses | Whether the state is reachable at all | Rare branches the persona never triggers |
| Debugging | Immediate. One turn to inspect | Archaeology across a whole transcript |
Next-reply earns its place through reachability. Asserting on behaviour at turn forty otherwise means driving thirty-nine probabilistic turns to arrive there, and the path differs every run. Writing the history directly pins the starting state so the test is about one decision rather than everything that led to it. Simulation then covers what pinning cannot: whether the agent reaches that state at all, and what it costs in turns.
Both rest on the same primitive, which is authoring the context window by hand rather than generating into it. That is the identical mechanism behind injecting a scripted utterance as an assistant turn, described in Deterministic Conversation.
Writing Criteria
A judge evaluating whether a response met a criterion is itself a model, with all the variance that implies. Criteria written as observable facts hold up; criteria written as impressions drift with every model change.
Helpfulness is an impression a judge scores differently across model versions. A stated fact before a stated event is checkable.
Well is doing all the work in the first version. The second names the behaviour that distinguishes good from bad.
Naming the fields and the ordering turns a judgement call into an observation.
Absence of hallucination is unfalsifiable as written. Traceability to a source is checkable line by line.
Anchoring a judge with a handful of labelled examples, some passing and some failing, does more for consistency than lengthening the criterion. Pin the judge model and its temperature, or the same transcript will pass and fail across a week with nothing having changed.
Asserting on Tool Calls
Tool behaviour is the most mechanically checkable thing an agent does, and the assertions divide into four kinds. The fourth is the one most suites omit.
| Assertion | Checks | Matching | Missed If Omitted |
|---|---|---|---|
| Selection | That the correct tool was chosen from those available | Exact | A plausible neighbouring tool used instead, producing a confident wrong answer |
| Parameter values | That arguments carry the values the conversation established | Exact where precision matters | Right tool, wrong account, which is worse than not calling at all |
| Extraction accuracy | That values were lifted correctly from natural speech into structured fields | Normalised | Transcription and formatting errors in dates, amounts and identifiers |
| Non-invocation | That a tool was NOT called when it should not have been | Exact | Actions taken before verification, the failure nobody writes a test for |
Tests Built From Real Traffic
The best test suite is a record of what has already gone wrong. Real conversations contain interruptions, accents, partial information and the specific confusions your users actually have, none of which a hand-written persona reliably invents.
Pick transcripts that failed, that a human had to rescue, or that took far more turns than they should have. Successful calls make weak tests.
Watch for: Selecting only failures builds a suite that says nothing about whether ordinary calls still work.
Real conversations carry personal data. A fixture is stored, copied and shared far more widely than a call recording, and outlives its retention policy.
Watch for: Redaction that changes the shape of an utterance changes what is being tested.
Decide what should have happened, which is usually narrower than everything that did. One conversation typically yields one or two criteria worth keeping.
Watch for: Asserting on the whole transcript produces a test that fails on any harmless rewording.
If the failure was one bad decision at an identifiable point, it becomes a next-reply fixture. If it was a conversation that never converged, it becomes a simulation.
Watch for: Turning every incident into a full simulation makes the suite slow and its failures hard to read.
Running Them Continuously
A suite of probabilistic tests behaves differently in a pipeline from a suite of deterministic ones. Every decision below has to be made explicitly or it will be made accidentally.
| Decision | The Question | Default That Usually Works |
|---|---|---|
| Gate or report | Does a failing agent test block a release, or raise a flag | Gate on tool and safety assertions, report on judged quality |
| Flake tolerance | How many runs must pass before a test counts as passing | Best of three for judged criteria, single run for deterministic ones |
| Suite cost | Every run spends real inference on both agent and judge | Full suite nightly, a fast deterministic subset on every change |
| Judge pinning | What happens when the judge model is upgraded | Pin the version, and rebaseline deliberately rather than discovering drift |
| Baseline drift | How you tell a real regression from ordinary variance | Track pass rate over time rather than treating a single run as truth |
Choosing a Shape
| You Want to Know | Use | Because |
|---|---|---|
| Does it make the right call in this exact situation | Next-reply | Pins the state so the assertion is about one decision |
| Can it get someone from opening to resolution | Simulation | Only a full run reveals loops, dead ends and turn cost |
| Does it call the right thing with the right values | Tool call | Deterministic and cheap, so it can run on every change |
| Did we break something we already fixed | Replay | The case is real and the expected behaviour already agreed |
| Does it stay safe under a hostile caller | Simulation | Adversarial behaviour needs turns to develop, so a pinned state will not show it |
Related: LLM Evaluation for benchmarks and judge design, Judge and Escalation for evaluation inside the runtime rather than in a suite, Turn-Taking for the interruption behaviours worth a scenario class of their own, Agentic Errors for the failure taxonomy a suite should cover, and Agentic Workflows for testing a graph rather than a single agent.