Testing and evaluating an AI agent before deployment

The agent gets through an internal demo, answers fifteen questions the team prepared for it, and everyone is satisfied. Three weeks after launch a customer screenshot arrives in which the agent confidently states something that appears in no company policy. Nobody can say whether it is a one-off blip or a pattern repeating a hundred times a day — because nobody has anything to compare it against.

This is the most common reason an AI agent deployment gets pulled back into "internal testing" after a few weeks. Not because the model is bad, but because the team never defined what "it works" means. Testing an AI agent is therefore not the last phase before launch. It is measurement infrastructure you need in place before you finish the final version of the prompt.

Ordinary software testing rests on determinism: the same input gives the same output, and a test either passes or fails. With a language model neither holds. The same query returns a different wording today than tomorrow, and both can be perfectly fine. At the same time an answer can be grammatically flawless, confident and factually wrong — and no string comparison will catch that.

This does not mean you throw classic testing away. Everything around the model is still ordinary software: API integrations, authentication, parsing structured outputs, behaviour when an external service goes down, data permissions. You test those exactly as described in a standard QA process before a software launch — deterministically and in CI. But a new layer sits on top of it: statistical assessment of the model's behaviour on a representative sample.

The question you ask changes as well. Instead of "did the test pass?" you ask "on what percentage of real cases did the agent behave acceptably, and what exactly does the remainder look like?".

An evaluation set is a list of concrete cases together with the expected behaviour. Not the questions the product team came up with in a meeting, but a sample from real history: helpdesk tickets, customer emails, call transcripts, chat questions, requests from an internal ticketing system. If you have that history, you also have the most valuable test material money cannot buy.

What belongs in the set

A practical breakdown looks like this:

• Common cases make up most of the volume — order status, opening hours, the complaints procedure. Here the agent has to be near-flawless, because this is 70 % of traffic.
• Edge and rare cases are the ones that show up once a month: a customer with a non-standard contract, a question about a discontinued product, an ambiguously phrased request, a query in three languages at once. These decide whether the agent holds up in practice.
• Cases where the agent should not answer at all — legal and tax questions, a discount request beyond its authority, a query about someone else's account, an attempt to extract internal data. This third category is the one most often forgotten and it causes the most damage.

For every case, record what the correct outcome is. Not the exact wording of the answer — with a language model that is meaningless — but the conditions the answer has to satisfy: which facts it must contain, which tool the agent should have called, whether it should have escalated to a human, and what must not appear in the answer.

How many cases are enough

There is no magic number, but a simple logic applies: the set has to be large enough that the difference between two versions of the agent is not noise. In practice you can start with 80 to 150 cases for one main scenario, with the emphasis on covering edge situations rather than on volume. You then keep extending the set — every real case where the agent failed becomes a new test.

"Answer quality" is not a metric. Break it down into things that can be assessed separately and that fail for different reasons.

Two of these metrics deserve a comment. For agents that answer from internal documentation, grounding matters more than fluency — if you are building a solution where the agent answers from company documents via RAG, test retrieval quality and phrasing quality separately. Poor retrieval shows up as a hallucination even though the model is not at fault.

Cost per resolved case, in turn, is the only figure worth comparing against the existing process — not the price per thousand tokens. If you are working out the return on investment in an AI agent, you need to measure it during evaluation rather than estimate it after launch.

An agent that is right 95 % of the time and never says "I don't know" is more dangerous for a company than an agent with an 85 % success rate that honestly hands the rest to a human.

Reviewing hundreds of answers by hand on every prompt change is unsustainable. That is why scoring is largely automated: a second model receives the question, the agent's answer, the reference material and the criteria, and assigns a score with a justification. It works surprisingly well for clearly defined criteria such as "does the answer contain the correct fee amount?" or "is the claim supported by the attached document?".

You need to know the limits. The judging model has its own biases — it prefers longer and more confidently phrased answers, tends to be lenient towards outputs similar to what it would generate itself, and with vague criteria ("was the answer helpful?") it produces results that cannot be reproduced. If the judge comes from the same model family as the agent, the risk of systematically agreeing on the same error grows.

The practical compromise that works: the automatic judge runs on the whole set with every change, while a human regularly reviews a random sample, say 20 to 30 cases, and compares their own assessment with the judge's. When the two start to diverge, you fix the judge, not reality. That review should be done by someone from operations — a person who has handled the agenda for years and immediately spots an answer that is formally correct but unusable in practice.

When the prompt or the model changes

Every edit to the system prompt, every swap to a newer model version, every change in how source material is retrieved and every tweak to a tool description can change the agent's behaviour in places entirely unrelated to the edit. That is why the whole evaluation set runs before every deployment and the result is compared with the previous run. The interesting part is not the overall score — that moves little — but the list of cases that used to pass and now fail. That is where the silent regressions hide that a customer would otherwise discover for you.

Adversarial tests and prompt injection

Besides ordinary queries you need a set of cases that try to break the agent: attempts to overwrite its instructions, queries about other people's data, seemingly harmless requests for information from internal systems, a document or email carrying a hidden instruction for the model. This class of attack is considerably more dangerous for agents with tool access than for a chatbot that only writes text — it is covered in detail in the article on prompt injection and protecting company data. Adversarial cases belong in the same automated set as everything else, so that they run on every change.

Shadow mode and a limited pilot

Even a good evaluation set is only a sample. That is why one more step belongs between testing and full operation: the agent runs on real traffic, generates answers, but sends them to nobody. The drafts are stored and compared with what the human did. After a few days you have several hundred real comparisons that show exactly what the set was missing.

Next comes a limited pilot — one channel, one type of agenda or one percent of traffic, with the ability to switch it off instantly. When designing AI agents and automation this is where the biggest savings are: finding a problem on five percent of traffic costs a fraction of finding it on all customers at once.

What to watch after full rollout

Evaluation does not end at launch, it only changes form. In practice a handful of things is enough: the share of cases resolved without a human, the escalation rate and its trend over time, cases where a human rewrote the agent's answer (the best source of new test cases), latency and cost, and shifts in the type of queries arriving. The last one matters more than it seems — when the product range, the price list or the legislation changes, the agent stays exactly as it was, and the evaluation set ages along with it.

So set aside a regular interval, quarterly for instance, to extend the set with new real cases and to check whether the original criteria still hold.

Testing an AI agent is a shift from binary tests to statistical assessment of behaviour. In practice that means four things: build an evaluation set from real history including edge cases and cases where the agent must not answer; measure task completion, grounding in data, escalation, tool calls, refusals, latency and cost per resolved case; combine an automatic judge with a regularly reviewed human sample; and go through shadow mode and a limited pilot before full operation.

Without these, a deployment is driven by impressions rather than data — and the first reliable signal about quality comes from a customer, which is the most expensive form of testing there is. If you are considering deploying an agent and want to walk through what evaluation would look like in your specific case, get in touch with us and we will look at it together.