How to Evaluate AI Agents: Know Yours Is Good Before Your Customers Do
You evaluate an AI agent the way you check a new hire's work. Write down what a good answer looks like. Collect 20 real cases with answers you would accept.
Run each case three times after every change. Then check a small sample of live work every week, and act when a warning number moves.
AI agent evaluation: stop grading the demo
Your agent answered 40 customer emails overnight. You read three over coffee. All three sound great.
Sounding great is not the job. Being right is. Nobody checked the other 37.
AI agent evaluation is how you find out which answer your customer actually got. It means testing the agent's work against answers you already trust, on a schedule. Think of it as quality checks for a new hire who never sleeps.
Here is the short version. Write down what good looks like. Build a small set of real test cases, run them more than once after every change, and check whoever does the grading.
Most teams skip all of it. They judge the agent by the demo, and the demo is the best day it will ever have.
A demo is a highlight reel. You need the game tape.
A vibe check grades the best day; an evaluation grades every day
| Vibe check | Evaluation |
|---|---|
| Reads a few outputs that sound good | Scores 20 real cases against a written rubric |
| Runs once, on the demo example | Runs each case three times after every change |
| Trusts whoever is grading | Tests the grader against people |
| Finds problems when customers do | Finds problems before release |
Our comparison. Evaluation as set out in this guide.
Write down what good means before you build anything
Decide how you will grade the work before you build the thing that does it. Chip Huyen calls this evaluation-driven development in AI Engineering. It is the AI version of writing tests before code.
Why first? Because good is harder to pin down than it looks.
LinkedIn hit this on a job-matching tool. An answer like "You are a terrible fit" can be true and still useless. The useful answer names the gap and how to close it.
So your first document is a one-page rubric. It says what the agent should do, what it must refuse, and what a pass and a fail look like. Give one real example of each.
The refuse part is the one teams skip. If nobody ever wrote down what a bad answer looks like, nobody will spot one.
Put the rubric inside the agent's job spec. Then the brief and the test describe the same job.
No rubric, no test. Just opinions with a login.
A rubric says what pass and fail look like, including what to refuse
Example: an agent that drafts replies to billing emails
| Criterion | Pass looks like | Fail looks like |
|---|---|---|
| Correct | Matches the refund policy page | States a refund rule we do not have |
| Useful | Tells the customer the next step | True, but leaves them stuck |
| Safe | Hands billing disputes to a person | Promises a credit on its own |
| Out of scope | Politely declines legal questions | Gives legal advice |
Example rubric, ours. Structure follows the evaluation guideline in Huyen, AI Engineering.
The 8-Minute Check reviews a full day of agent work without redoing it: diff what the agent touched (1 minute), check the tripwire numbers (1 minute), read its plan instead of the transcript (2 minutes), spot-check one routine, one edge and one high-stakes case against the rubric (3 minutes), and confirm the machine-checked must-always rules (1 minute). It sits on top of a 20-case golden set that is rerun three times after every change.
Build a golden set of 20 real cases
A golden set is a fixed list of real inputs, each with an answer your best person signed off on. You rerun it after every change to the prompt, the model or the tools. Software teams call this regression testing.
Start small. Five to 20 examples is enough to begin, and you read every output yourself.
The team behind one of the first AI coding assistants built its test before the product. In Prompt Engineering for LLMs, John Berryman and Albert Ziegler say that early test is why they could move so fast.
Where do the 20 come from? Your own records. Last month's tickets, sent emails, approved drafts.
Mix them on purpose. We use 12 routine cases, 4 edge cases, 2 requests the agent should refuse and 2 with typos or messy input.
Then hide one duplicate. Put the same case in twice under different numbers. If it gets two different scores, your grader is noisy.
The golden set is also what makes switching models safe. A model swap breaks more things than any prompt edit.
Twenty real cases beat a thousand opinions.
Start with 20 cases you read yourself, then automate the scoring
- Example suite: 5 to 20 cases You read every output after each change and note what went wrong.
- Golden set: 20 to 100 cases Each case has an accepted answer and is scored pass or fail on the rubric.
- Harness: hundreds of cases A script or a checked AI grader scores every case on every change.
Stages adapted from Berryman and Ziegler, Prompt Engineering for LLMs, chapter 10.
Example: most of a 20-case golden set is routine work, on purpose
Our default mix of cases, plus one hidden duplicate to test the grader
- Routine cases 12
- Edge cases 4
- Should refuse 2
- Typos or messy input 2
- Hidden duplicate 1
Our recipe for a first golden set. Adjust the mix to your own work.
A working golden set: real cases, answers from your best person, three runs each
- Your best person writes the answers. Never the AI.
- Write down what a bad answer looks like, or nobody will catch one.
- One failed run fails the case. This is the one to fix.
- The hidden duplicate. Two different scores means the grader is noisy.
- Rerun all of it after every change.
Example page built for this article with a made-up brand, 2026-10-18. The 30-day window and 5 to 7 day timeline are example policy, not a real company's.
Run every case three times, not once
An agent can answer the same question differently on two runs. So one pass shows it can do the task. It does not show it will.
Researchers behind tau-bench tested leading agents on realistic customer service tasks. The best ones finished fewer than half. Asked to get the same retail task right eight times in a row, they managed it less than 25 percent of the time.
The math is simple and brutal. An agent that passes 75 percent of the time passes three runs in a row only about 42 percent of the time.
Long tasks multiply the same way. At 95 percent per step, a 10-step task lands about 60 percent of the time. A 100-step task lands well under 1 percent.
Your customer does not meet your average run. They meet whichever run they get.
Our rule: run each golden case three times after every change. A case passes only when all three runs pass.
Repeat runs cost money, so budget them like any other AI spend. They cost less than one public mistake.
How to ship a broken agent in four easy steps:
1. Test it once, on the example from the sales demo.
2. Let it grade its own homework.
3. Change the prompt on Friday and skip the rerun.
4. Call the first angry customer an edge case and hope like hell.
One good run is a demo, not a test.
At 95 percent per step, a 10-step task succeeds about 60 percent of the time
Chance the whole task succeeds, by number of steps, when each step succeeds 95 percent of the time
- 95% per step: 1 95, 5 77.4, 10 59.9, 20 35.8, 50 7.7, 100 0.6
Our arithmetic (0.95 to the power of the step count). The 10 and 100 step figures also appear in Huyen, AI Engineering.
An agent that passes 75 percent of the time passes three in a row 42 percent of the time
Chance of passing every run, for an agent that passes 75 percent of single runs
- Pass one run 75
- Pass two in a row 56.3
- Pass three in a row 42.2
Our arithmetic: 0.75, 0.75 squared, 0.75 cubed.
Leading agents finished fewer than half the tasks, and far fewer when asked to repeat
- A test of AI agents on realistic customer service conversations.
- pass^k means: does the agent get the same task right k times in a row.
- Leading agents finished fewer than half of the tasks.
- Asked to repeat a retail task 8 times, they got all 8 right less than a quarter of the time.
- Consistency, not one good run, is the real test.
Screenshot of the tau-bench abstract (Yao et al.) on arXiv, taken 2026-10-18. Model name blurred.
Name the failure, then count how often it happens
A pass rate tells you something broke. It does not tell you what. A smoke alarm does not tell you which room is on fire.
So sort every failed case into a type, and count each type. Huyen's AI Engineering gives a clean list of agent failures, and the table below is our plain version of it.
The sneakiest type is the false finish. Ask an agent to put 50 guests in 30 rooms, and it may place 40 and report the job done.
You catch that with a log. Record every tool call the agent makes and what came back, then actually read it.
Count the types every week. The biggest count is your next fix. It is often a tool or an instruction, not the model.
You cannot fix a percentage. You can fix a type.
Six failure types, and the cheapest way to catch each one
| Failure type | What it looks like | How to catch it |
|---|---|---|
| Wrong tool | Calls a tool it does not have | Tool call log |
| Wrong inputs | Right tool, wrong or missing values | Tool call log plus golden answers |
| Wrong goal | Solves a different task or breaks a limit, like budget | Rubric check |
| Too slow | Finishes after the answer stopped mattering | Time per task |
| False finish | Says done when it is not, like 40 of 50 guests placed | Script that counts the result |
| Bad tool data | The tool itself returns wrong output | Test each tool on its own |
Failure types from Huyen, AI Engineering, agent evaluation section, in our words.
How many test cases do you need to trust a change?
For a big gap, fewer than you think. For a small gap, far more.
A rough guide in AI Engineering: about 10 cases to be sure of a 30-point gap, about 100 for 10 points, about 1,000 for 3 points. Each time the gap shrinks to a third, you need ten times the cases.
So 20 cases can catch a big break. They cannot prove a new prompt is 4 percent better.
Two cheap checks keep you honest. First, redraw your set at random with repeats allowed, score each draw, and watch the swing. Statisticians call it a bootstrap.
If one draw says 90 percent and another says 70, your set is too small to decide anything.
Second, report a range, not a point. Douglas Hubbard and Richard Seiersen show the method in How to Measure Anything in Cybersecurity Risk. One success in six tries really means somewhere between about 5 and 52 percent.
Our decision rule: ship the new version only when it wins by more than the swing you saw in the redraws. A tie goes to the version you already trust.
Small set, big claims? That is not data. That is a mood.
Each time the gap shrinks to a third, you need ten times the cases
Rough number of test cases needed to be 95 percent sure one version beats another
| Gap you want to detect | Cases needed | What it means for a 20-case set |
|---|---|---|
| 30 points | About 10 | Clear |
| 10 points | About 100 | Too small |
| 3 points | About 1,000 | Far too small |
| 1 point | About 10,000 | Not a question for 20 cases |
Huyen, AI Engineering, Table 4-7 (values credited there to OpenAI). Right column is ours.
One pass in six tries could mean anything from 5 to 52 percent
- What 1 pass in 6 looks like 16.7
- Low end of the 90 percent range 5.3
- High end of the 90 percent range 52
Hubbard and Seiersen, How to Measure Anything in Cybersecurity Risk, beta distribution example.
Can an AI grade an AI? Yes, if you grade the grader
An AI grader is a second model that scores the first model's work against your rubric. It lets you score hundreds of outputs without hiring hundreds of reviewers.
It works better than most people expect. In a 2023 study, strong models used as judges agreed with human raters over 80 percent of the time. That is about as often as two humans agree.
It also has bad habits. Graders tend to favor the answer shown first, the longer answer, and answers from their own model.
In one study, graders picked longer answers with factual errors over shorter answers that were right.
A grader can also drift while nobody watches. Fix a typo in its prompt and the score can jump two points while the agent stays exactly the same.
Self-grading is the worst case. Models grade more fairly when they think the work belongs to someone else.
So test it like a new reviewer. Have three people score the same 30 outputs, then add the AI as a fourth scorer. If it agrees with them about as well as they agree with each other, keep it.
People are noisy too. Radiologists shown the same 96 x-rays a week apart changed their call 23 percent of the time.
That is why the hidden duplicate matters. It measures any grader, human or AI.
Never trust a grader you have not graded.
AI graders have four known habits, and each has a fix
| Habit | What happens | Fix |
|---|---|---|
| First-position bias | Favors the answer shown first | Shuffle the order and score twice |
| Length bias | Prefers longer answers, even wrong ones | Score correctness before style |
| Self-preference | Scores its own model's work higher | Never tell it the work is its own |
| Drift | Score moves when its prompt is edited | Version the grader prompt like code |
Huyen, AI Engineering; Berryman and Ziegler, Prompt Engineering for LLMs. Fixes in our words.
Human experts disagree with themselves often, so measure every grader
- Radiologists who changed their call on the same x-rays a week later 23
- Expert estimate pairs on hidden duplicates that differed by over 10 points 26
- How often strong AI judges agreed with human raters 80
Hubbard and Seiersen, How to Measure Anything in Cybersecurity Risk; Zheng et al. 2023 (arXiv).
The 8-Minute Check: review a day of agent work without redoing it
When an agent does a full day of work, nobody can redo it to check it. So the review tests your safety checks, not the work itself.
The 8-Minute Check is five moves in a fixed order.
Diff the surface (1 minute): compare before and after, and list what it created, changed or touched. Anything outside its scope is an instant flag.
Check the tripwires (1 minute). A tripwire is a number that alerts a person when it moves, like spend, errors or escalations.
Read the plan, not the transcript (2 minutes). Skim the agent's stated approach the way you skim a report summary.
Spot-check three items (3 minutes): one routine, one edge case, one high-stakes case, graded on the rubric.
Confirm the must-always rules (1 minute). Nothing sent outside, totals that add up, tests that pass. A script checks them and you confirm.
It only works on top of the golden set. Without one, eight minutes is just a nice feeling.
Good managers have always worked this way. The tools changed. The job did not.
Five moves check a day of agent work in eight minutes
- 1. Diff the surface (1 min) What it created, changed or touched. Outside scope is a flag.
- 2. Check the tripwires (1 min) Spend, errors, escalations, odd tool calls.
- 3. Read the plan (2 min) The stated approach, not the full transcript.
- 4. Spot-check three (3 min) One routine, one edge, one high-stakes case, on the rubric.
- 5. Confirm the must-always rules (1 min) Nothing sent outside, totals add up, tests pass.
The 8-Minute Check, our review routine.
Treat a great result as a bug until you rule it out
A big jump after a change should make you suspicious, not happy. Surprising numbers usually come from a mistake in how you measured.
Atul Gawande tells a story about this in The Checklist Manifesto. His team's surgical checklist trial showed deaths down 47 percent, and he went hunting for errors before he believed it.
He asked three questions.
Did the cases get easier? Did being watched change behavior, the Hawthorne effect? Did it only work in some hospitals?
He trusted the result only after all three answers came back no. Ask the same three about your agent's best week.
Did this week's tickets get easier? Did your team review harder because they knew you were looking? Did the gain show up for one client only?
Then run the same version against itself. Ron Kohavi, Diane Tang and Ya Xu explain this A/A test in Trustworthy Online Controlled Experiments. If two identical setups look different, your test is broken, not your agent.
When you compare an old and a new version, look closest at the cases where their answers differ. Same discipline as testing ad creative.
A win you never tried to break is a rumor.
Three questions to ask before you believe a big jump
| Question | In the surgical trial | For your agent |
|---|---|---|
| Did the cases get easier? | No. Slightly more emergencies, same mix | Were this week's tasks simpler? |
| Did being watched change behavior? | No. Watched and unwatched rooms improved equally | Did reviewers try harder this week? |
| Did it only work in some places? | No. All eight hospitals improved | Did the gain come from one client? |
Trial checks from Gawande, The Checklist Manifesto. Agent column is our reading.
Does a small team need this? Ask Air Canada
You may think this is only for companies with an AI team. Look at who got caught.
Air Canada's website chatbot once told a grieving customer he could claim a bereavement discount after flying. He could not.
The airline argued the bot was responsible for its own words. A tribunal rejected that and ordered it to pay $812.02 in Canadian dollars, as CBC also reported.
Cheap in cash. Expensive in headlines.
New York City's MyCity chatbot was built to help small businesses. Reporters later caught it telling owners to do things that break the law.
Neither failure needed an engineer to catch it. It needed someone to ask the bot 20 real questions and read the answers.
Marketing agents fail in public, so their tests matter even more. The table below shows what to check for content, outbound and ad copy, with buyer words pulled from your customer truth system.
Decide who signs off on each agent in your AI governance framework. Whatever your agent says, you said it.
Each marketing agent needs its own test and its own checker
| Agent | The test | Who checks |
|---|---|---|
| Content | Every number and quote appears in the source notes; a person names one new point | Script, then a person |
| Outbound | 19 of 20 hand-checked contacts match on name, title and company; drafts stop before sending | Script |
| Ad copy | Every claim sits next to its proof, or it is cut | Script plus a person |
| Ad copy | Uses at least two phrases buyers actually said | AI grader, checked monthly |
Our checks. The FTC's small-business advertising guide is why proof comes before the ad runs.
Run it with AI this week
Paste the prompt below into your preferred AI. It asks about one agent, then drafts your rubric, your golden set plan and your pass rule.
It will not write your accepted answers. Those come from your records and your best person, and that is the point.
1. Pull 20 real past cases and have your best person write the accepted answer for each.
2. Run each case three times on the current agent. Log which cases pass all three.
3. Score 30 outputs with two colleagues, then check whether your AI grader agrees with you.
Tests in place? Next, wire them into your agent harness so they run on every change. Or have us build the system with its tests already in place.
Remember the 37 emails nobody read? Trust the agent that already failed your tests in private. Not the one that fails your customers in public.
Paste into your preferred AI. It asks about one agent, then drafts your rubric, golden set plan, pass rule and grader check.
Act as an AI quality reviewer. Help me build a first evaluation for one AI agent. Ask me one question at a time, with the example: 1. What does the agent do, and for whom? (Example: drafts replies to customer billing emails) 2. What must it never do? (Example: promise refunds, give legal advice) 3. Who on my team knows this work best? (Example: our support lead) 4. Where can I find 20 real past cases? (Example: last month's closed tickets) 5. What change is coming next? (Example: a new model next month) Then write, in plain words: A. Rubric: 3 to 5 criteria, each with one pass and one fail example. Include one rule for out-of-scope requests. B. Golden set plan: 20 real cases (12 routine, 4 edge, 2 out of scope, 2 messy input) plus 1 hidden duplicate. Never write the accepted answers yourself. Mark each one "to be written by" the person from question 3. C. Run rule: run every case 3 times after any prompt, model or tool change. A case passes only if all 3 runs pass. D. Decision rule: 20 cases only show big gaps, around 30 points. Smaller gaps need more cases. A tie keeps the current version. Report pass rates as a range, not one number. E. Grader check: 3 people score the same 30 outputs, then the AI grader scores them as a 4th. Keep the AI grader only if it agrees with them about as well as they agree with each other. Shuffle the answer order and never tell it the work is its own. F. Failure log with these types: wrong tool, wrong inputs, wrong goal, too slow, said done when not done, tool returned bad data. G. A daily 8-Minute Check for this agent. Never invent numbers about my business. Ask me for them. End with: Next three actions: 1. 2. 3.
Filip Kovacevic is the founder of Anar Agency in Phoenix. He spent five years running SEO, Meta ads, email, and content for medical practices across the United States, and now does the same work for healthcare, home-service, legal, and professional-service businesses, alongside AI implementation. More from Filip →
Talk it through in 15 minutes
Bring one question from this article. We will tell you what we would do, whether or not you hire us.
Book a 15-Min Chat →