Blog: How to Evaluate an AI Agent Before It Reaches Production
Most guidance on testing AI agents is written for engineers. The people who decide whether an agent goes live are usually someone else: the owner of the process, the head of operations or the CTO. What they need is evidence they can put their name to. This article is written for that decision. It covers what to ask for before an agent touches real work, how to read the results, and who should have the authority to stop the agent afterwards.
Where there are facts to cite, such as published research and vendor announcements, they are linked. The rest is our own view of how to run the decision, and it is marked as such.
The release decision belongs to the business
An evaluation produces evidence. Someone then has to turn it into a yes or a no, and that person should be the owner of the process the agent will work in, because they carry the consequences. Engineers can say how often the agent was right. Only the process owner can say what a wrong result costs, whether it can be undone, and how much of it the business can accept.
In our view, three roles should be named before any testing starts:
The release owner, from the business, who approves going live and sets the threshold.
The technical owner, who can stop the agent at once and who runs the tests.
The reviewer, who reads a sample of the agent's real work every week after launch.
If the first role is empty, the test results have nobody to go to, and the decision gets made by default.
Start with what a wrong action costs
How much evidence is enough depends on what the agent is allowed to do. We find it useful to sort agents into four levels before deciding on the tests.
Level | What the agent does | Example | Evidence to ask for before release |
1. Reads and summarises | Reads records and reports what it finds | Summarising a set of documents for a person | Accuracy on real cases, checked by the process owner |
2. Drafts for review | Prepares work that a person approves | A draft reply or a proposed entry | Accuracy, and how often reviewers change the draft |
3. Acts, and it can be undone | Changes records in a way that can be reversed | Updating a status field | Results over repeated runs, and a tested way to reverse a change |
4. Acts, and it cannot be undone | Pays, sends outside the company or deletes | Releasing a payment or emailing a customer | Very high consistency over repeated runs, hard limits and human approval before the action |
The level sets the bar. An agent at level 4 needs far more evidence than one at level 1, and the same test results can justify one release and not the other. A team can move an agent up a level once it has results at the level below, and starting a new agent at level 1 or 2 is the cheapest way to collect real evidence.
Build the task set from the process itself
The test cases should come from the process, and the people building the agent should not be the ones inventing them. Collect real past cases, including the ones that went wrong, and have the process owner write down the correct outcome for each. Anthropic's engineering team suggests that 20 to 50 simple tasks drawn from real failures is a good start and that teams do not need to wait for hundreds (Demystifying evals for AI agents, 9 January 2026).
The set should include the awkward cases as well as the routine ones:
Inputs that are incomplete, scanned or formatted differently from usual
Two sources of data that disagree
Requests outside the agent's job
Cases where the right answer is to hand over to a person
Keeping a share of the cases aside, and not using them while the agent is being adjusted, gives a fairer reading at the end. Cases the agent was tuned on will flatter it.
Score what the agent did, over several runs
The check that matters most is the end state in the systems the agent touched, and what the agent said about its own work comes second. The τ-bench research benchmark scores agents this way, by comparing the database at the end of a task with an expected state (arXiv 2406.12045).
Each case should also be run more than once, because the results vary. The τ-bench authors reported that the function-calling agents they tested in 2024 completed fewer than half of the tasks, and that success across eight repeats of the same task fell below 25% in the retail domain. Those were the models of that time. The arithmetic still holds for any agent: one that succeeds on 75% of single runs succeeds on three runs in a row about 42% of the time, which is the example Anthropic gives. For an agent that acts without review, the second figure is the one the business will experience.
Rules that can be written down, such as a total that must match or a field that must be present, are best checked with code. Judgement calls can be scored by a model, with people reviewing a sample. Whoever scores, someone should read the transcripts of the failures, because a failure sometimes comes from the scoring and not from the agent. OpenAI's guidance on evaluating agent workflows starts from traces for the same reason (Evaluate agent workflows).
One set of results, two different decisions
The numbers below are made up to show the reasoning. They are not data from any real test.
Suppose an agent is tested on 40 cases, and each case is run five times. On 34 cases it is right every time. On 4 cases it is right in some runs and wrong in others. On 2 cases it is never right. That means 34 of 40 cases, or 85%, are handled correctly on every run.
If the agent drafts work for a person to approve (level 2 in the table above), those results could support a release. Reviewers would be told about the 4 uneven cases, and the 2 failures would go back for fixing. If the same agent releases payments (level 4), 6 cases in 40 with an unreliable or wrong result is far from enough, and those cases show where hard limits and human approval are needed.
The results are identical in both situations and the decisions are opposite. That is why the threshold has to be set by the release owner before the results arrive. A threshold chosen afterwards tends to be whatever the results happen to meet.
Test limits, permissions and stopping
An agent that does its tasks well still has to be tested on what it must not do. Four checks cover most of it:
Scope. Give the agent requests outside its job and confirm that it declines or hands over.
Permissions. Confirm it can reach only the systems and records its role needs, and that its access can be withdrawn quickly.
Spending and volume limits. Set the limit in the platform or the surrounding system, in a place the agent cannot edit. Then test that requests stop when the limit is reached and that the agent cannot raise it.
Stopping. Stop the agent in the middle of a task and look at what is left. Some work will be complete, some half done and some waiting for approval, and someone has to be able to tell which is which and finish it.
The last check is often skipped because it is awkward. It also settles who has the authority to stop the agent and who decides how the interrupted work resumes. Those can be different people, and it is better to agree on them before launch than during an incident.
Vendors are adding controls of this kind. Salesforce announced on 27 October 2026 a generally available Agent Kill Switch and a Model Wallet that sets hard spending limits and stops requests before a budget is exceeded (Salesforce announcement). Controls like these still have to be tested in the company's own setup.
Keep evaluating after launch
Pre-launch tests cannot cover every case the agent will meet. Production monitoring shows real behaviour at scale and catches problems that test cases missed. Anthropic's engineering team notes its limits: it is reactive, because problems reach users first, and it needs instrumentation. Tests before release and monitoring after release cover different gaps, so both are needed.
A few habits keep the evaluation useful:
Add every real failure to the task set, so the same mistake is tested for from then on.
Run the full set again whenever the model, the prompts or the tools change.
Have the reviewer read a sample of real work each week, including the cases that went well.
Track the cost of a completed task next to accuracy.
The last habit has research behind it. A Queen's University study followed 35 releases of one open-source coding tool with the underlying model held fixed. It found no statistically significant improvement in benchmark results, while later releases used nearly twice the tokens and tool calls (arXiv 2607.03691, version dated 20 July 2026). The study covers one tool and one benchmark, so it is not a general rule. It does show that a newer version can cost more without performing better, and that this is only visible when cost is measured.
Ways an evaluation misleads
A good-looking result can still be a weak one. These are the situations we would check for first:
The cases were also used to adjust the agent, so the results flatter it.
The builders defined what "correct" means, so ambiguous cases were settled in the agent's favour.
One average score hides a category where the agent is poor, such as the exceptions.
Each case was run once, so the variation between runs was never seen.
Nobody read the failures.
Cost per task was never measured.
The tests were not run again after a change of model, prompts or tools.
A scorecard for the release meeting
These seven questions can be put to the team before an agent goes live. A release owner who cannot get a clear answer to one of them has found the gap to close first.
Question | Evidence to ask for | Who answers |
What is the worst action the agent can take, and can it be undone? | The action level from the table above | Release owner |
How many real cases were tested, and who wrote the expected outcomes? | The task set and the process owner's sign-off | Release owner |
How often was the agent right across repeated runs? | The pass rate and the all-runs rate, by case type | Technical owner |
What does it do on exceptions and out-of-scope requests? | Results for those cases on their own | Technical owner |
What does one completed task cost? | Cost per task, next to accuracy | Technical owner |
Who can stop it, and what happens to half-finished work? | The stopping test and two named people | Both |
Which result after launch triggers a rollback? | A written threshold | Release owner |
A note on how we would approach it
Kreeda Labs builds AI systems that work inside business processes. For an agent, we would build the task set with the process owner from the company's own cases, deliver the evaluation set and the scoring rules together with the system, and set them up so the client's team can run them in its own environment after handover. That keeps the evidence with the company and lets it be repeated whenever the model, the prompts or the tools change.
Summary
An agent is ready for release when the evidence matches the cost of its worst possible action. That evidence comes from real cases, repeated runs and checks on limits and stopping, and it should be read by the person who owns the process.
This article is general information about evaluating AI agents. Teams should adapt the practices to their own processes, systems and risk requirements.
Comments