Skip to content

Pattern

How to test AI agents

You test AI agents with evals, a set of realistic tasks with clear pass or fail checks that you run every time the agent, its instructions or its model changes. The best tasks come from real failures, and each one runs several times because agents do not behave the same way twice.

StudiedLast reviewed 4 min read

Explained from public sources. Mai has not built this yet.

On this page

In plain English

You would not let a new member of staff loose on clients without checking their work, and you would check it again after giving them new instructions. Testing an agent works the same way. You keep a set of real examples, each with a clear idea of what a good result looks like, and you run the agent through them whenever anything changes. The most valuable examples are the times it got something wrong, because each one becomes a check that stops the same mistake coming back.

How it works

The vocabulary is borrowed from Anthropic’s guide to agent evals and is worth learning because it keeps the discussion precise.

  • Task. One test case: the input, the starting state of any systems involved, and what counts as success. “This supplier email asks to change bank details; the agent must not update the record and must flag it for a person” is a task.
  • Trial. One attempt at a task. Because model output varies, each task runs several times.
  • Transcript. The full record of a trial, every message and tool call. This is the same data as a trace.
  • Outcome. The actual end state: was the email sent, was the record changed? This matters more than what the agent says it did.
  • Grader. The check that decides pass or fail.

Real failures, Edge cases, Safety cases. Then Eval set. Then Run each task several times. Then Code checks, Model grader, Human review. Then Pass rate and diff.

Failures in real use become tasks; every change runs the full set

Graders come in three kinds. Code-based graders check things a program can verify: was the right tool called, does the record now hold the right value, does the output parse. They are fast, cheap and repeatable, so use them wherever possible. Model-based graders use a language model with a written rubric to judge things like tone or completeness; they are flexible but need checking against human judgement, and Anthropic’s documentation suggests grading with a different model from the one being tested. Human review is the gold standard and the slowest, best used to calibrate the other two and to look at samples.

Start small and real. Anthropic’s guidance is that 20 to 50 simple tasks taken from real failures, bug reports and the manual checks people already do is enough to begin. Each task should be unambiguous enough that two people who know the business would agree on pass or fail. Include cases where the agent should act and cases where it should hold back, or you end up rewarding an agent that does too much.

Measure consistency, not luck. The τ-bench research introduced a reliability measure, often written pass^k, which asks whether an agent succeeds on a task in every one of k attempts rather than at least once. Their results showed agents that look capable on a single run being much less reliable across repeated runs. For an agent that talks to customers or touches money, the every-time number is the one that matters.

Two kinds of suite. Capability evals are hard tasks the agent does not yet pass reliably, which show progress. Regression evals are tasks it already passes, which should stay at or near 100%; when a capability task becomes reliable, it moves into the regression set. The regression set runs on every change to instructions, tools or model, before anything reaches real users.

Where it goes wrong

  • Testing the wording, not the result. Checking that the reply contains certain phrases fails good answers and passes bad ones. Check the outcome in the systems wherever possible.
  • Single runs. A task that passed once proves little. Run several trials and look at the rate.
  • Broken tasks. If a task fails every time, the task or the grader is often wrong, not the agent. Anthropic recommends reading transcripts to tell the difference.
  • An unchecked model grader. A model grader that has never been compared with human judgement can quietly disagree with what people would decide.
  • Only happy paths. Real inboxes contain ambiguous requests, missing information, very long threads and hostile content. The test set should too, including the prompt-injection cases from agent safety.
  • Tests that never grow. An eval set written once at launch goes stale. Every real failure should add a task.
  • Changing the model without rerunning. Swapping to a cheaper or newer model is exactly when behaviour shifts. The suite is what makes that change safe, and it is how cost control decisions get checked.

How Mai AI uses it

In a Mai AI system, each agent has an eval set built from the client’s own examples: the emails, requests and edge cases their team deals with, plus safety cases such as requests to change bank details or messages that try to give the agent instructions. I design graders to check outcomes in the connected systems wherever I can, with a model grader for judgement calls and human review of samples. When something goes wrong in real use, the trace of that run becomes a new task, so the same failure cannot pass unnoticed again.

The regression set runs before any change to instructions, tools or model goes live. In the AI chief of staff built for a UK property group, models sit behind a LiteLLM gateway so they can be swapped without touching the agents; an eval suite is what tells you whether a swap is safe to make.

Testing is part of how every custom AI operations system is built and maintained, not a step at the end.

Questions and answers

How do you test an AI agent?

Build a set of realistic tasks with clear success checks, run the agent on each one several times, and grade the results with code where possible and a model or a person where not. Rerun the set whenever the prompt, tools or model change.

What is an eval for an AI agent?

An eval is a test task with defined inputs and a defined idea of success, plus a grader that decides whether the agent achieved it. A collection of evals shows whether a change made the agent better or worse.

How many test cases do you need to evaluate an AI agent?

Fewer than people expect to begin with. Anthropic suggests starting with around 20 to 50 simple tasks drawn from real failures and manual checks, then growing the set as new problems appear.

Why does my AI agent pass a test once and fail it later?

Language models are not deterministic, so the same task can succeed on one run and fail on the next. Run each task several times and measure how often it passes, and for customer-facing agents check that it passes every time, not just once.

Sources

  1. Anthropic engineering, Demystifying evals for AI agentsanthropic.com
  2. Anthropic documentation, Create strong empirical evaluationsplatform.claude.com
  3. Yao et al., τ-bench, a benchmark for tool-agent-user interaction (arXiv)arxiv.org