AI Operations: Most AI Problems Are Operations Problems in Disguise
Most problems blamed on AI are operations problems with a new name. Scattered data, no clear owner, no way to check quality, and steps nobody needs. Test each one: would it still exist if AI were a fast new hire?
If yes, fix the process first. Then almost any AI tool works better.
Most AI problems are operations problems in disguise
Ask a room of executives what they want from AI.
Customer answers in seconds. Workflows that run themselves. Reports that write themselves.
Then ask what is stopping them.
Our data is a mess. Nobody owns this. We cannot tell good output from bad.
Read that second list again. Not one item is about AI. Not one.
That is the core of AI operations. Most AI problems are old operations problems with a new label: data, ownership, quality checks and process.
AI did not create them. It exposed them. It touches every team at once and works faster than your checks can keep up.
So here is the rule we work by. AI moves only as fast as the operations underneath it.
The good news? You already know how to fix operations.
What leaders want from AI is about AI; what blocks them is not
| What they want | What blocks it |
|---|---|
| Workflows that run themselves | Our data is all over the place |
| Customer answers in seconds | Nobody owns this |
| Faster engineering | We cannot tell if the output is good |
| Reports that write themselves | Everyone is too busy |
| Less manual work | We are afraid of what people will paste into it |
Illustration of a typical AI planning meeting. Not survey data.
The Costume Test: swap "AI" for "a fast new hire"
The Costume Test is a one-line check for any AI question. Replace the word AI with "a very fast new hire" and read the question again.
Does it still make sense? Then it is an operations question. You already own the playbook.
How do we know their work is good? That is quality control.
Who manages them? That is ownership.
What are they allowed to see? That is access rules.
What does each piece of their work cost? That is unit cost.
Where does their information live? That is data, and we cover it in how to make AI work across scattered data.
Questions that fail the swap are the new ones. Which AI vendor to bet on. How to check answers that come out different every run.
Run the test on your own list and count. Our bet: the new pile is the short one.
Old problems. New invoice.
Swap AI for a fast new hire and most questions become old ones
| The AI question | Asked about a fast new hire | Who already owns the answer |
|---|---|---|
| How do we know the AI output is good? | How do we know their work is good? | Quality control |
| Who manages the AI? | Who manages them? | Ownership |
| What can the AI access? | What are they allowed to see? | Access rules |
| What if the AI exposes client data? | What if they share client data? | Security policy |
| What does each AI task cost? | What does each piece of their work cost? | Unit cost |
| Which AI vendor should we bet on? | Makes no sense | A real AI question |
The Costume Test, our method.
Three steps sort any AI question in under a minute
- 1. Write it as asked Copy the question exactly as your team says it.
- 2. Swap the words Replace AI with "a very fast new hire" and read it again.
- 3. Route it Still makes sense? Give it to an operations owner. Breaks? Give it technical time.
The Costume Test, our method.
The Costume Test is a one-line check for any AI question: replace "AI" with "a very fast new hire" and read it again. If the question still makes sense, it is an operations question, so fix it with ownership, a written standard, fewer steps or a setup that makes the mistake impossible. Find the slowest step before you automate anything.
Save technical time for the questions that only make sense for AI.
Most failures come from not using what you already know
Most failures in mature fields do not come from missing knowledge. They come from not using knowledge you already have.
Two philosophers, Samuel Gorovitz and Alasdair MacIntyre, split failure into two kinds. Ignorance means nobody knows how yet. Ineptitude means the know-how exists and we fail to apply it.
Surgeon Atul Gawande builds The Checklist Manifesto on that split. His point: in medicine, the second kind is now as big a struggle as the first.
He also shows why good practice spreads so slowly. In one study, doctors took 17 years to adopt nine major new treatments for at least half of patients.
The cause was rarely laziness. The know-how had never been turned into a simple form people could use.
Sound familiar? Your best person knows what a good client reply looks like. Nobody wrote it down, so the AI cannot follow it either.
Our reading: that is an ineptitude problem wearing an AI costume. A smarter model will not fix it. A one-page standard will.
Where does your company sit on this? Our AI maturity assessment scores it in a few minutes.
If the know-how exists, the fix is operations, not a smarter tool
| Ignorance | Ineptitude |
|---|---|
| Nobody knows how yet | The know-how exists |
| Fix: research and new tools | Fix: make the known way easy to follow |
| AI example: a task too hard for any model | AI example: a good reply exists, but nobody wrote it down |
Categories from Gorovitz and MacIntyre, as told in The Checklist Manifesto by Atul Gawande. AI examples are ours.
Proven treatments took 17 years to reach half of patients
- Major treatment discoveries in the study 9
- Average time to reach at least half of US patients 17
- Time a Boeing team took to turn a crash finding into a tested checklist About 2 weeks
Source: The Checklist Manifesto by Atul Gawande, citing a medical adoption study.
Find the one step that slows everything down
A workflow moves only as fast as its slowest step. Speed up any other step and the whole thing barely moves.
Operations people call this the constraint. In Clockwork, Mike Michalowicz tells of a car bumper plant with parts piled up in front of the welders.
The welders were not slow at welding. Watched closely, they welded only about 10 percent of the time. The rest went to fetching, clamping, cleaning and carrying.
The fix was not a faster torch. A few teenagers were hired to stage the parts, and the pile was gone within days.
Here is the AI version. A team buys a tool that drafts proposals in two minutes. Every proposal still waits a week for one manager to sign it.
Did the tool speed anything up? Not for the client. The week of waiting is the constraint.
Two-minute drafts. One-week wait. The client feels the week.
Systems scientist Donella Meadows saw the same mistake between countries. In Thinking in Systems, she notes that rich nations send capital or technology, then wonder why nothing grows.
Nobody asked whether those were the real limits. AI is technology too. Check that it is your limit before you buy more of it.
The welders were welding only about 10 percent of the time
Share of the welders' working time in one car bumper plant
- Welding 10
- Fetching, clamping, cleaning, carrying 90
Example from Clockwork by Mike Michalowicz. The 90 percent is the rest of his figure of about 10 percent.
A faster tool cannot beat a weekly approval meeting
| Step | Before the new tool | After the new tool |
|---|---|---|
| Set up new infrastructure | Weeks or months | Minutes or hours |
| Get each change approved | A board that meets weekly | A board that meets weekly |
| How fast work reaches customers | Weekly at best | Weekly at best |
Example from Team Topologies by Matthew Skelton and Manuel Pais.
Simplify the steps before you automate them
Automation makes a process faster. It does not make it right.
Andy Grove ran operations at Intel and wrote the method in High Output Management. Draw every step of the process and count them. Set a target to cut 30 to 50 percent, then question why each step exists.
Across a wide range of Intel's office work, steps dropped by about 30 percent. That came from cutting steps, not from adding machines.
He also tells of an embassy buried in visa forms. Its fixes, like drop boxes and sorting forms by speed, only made the lines worse.
Why? They added work and never sped up the processing itself. His fix: stop checking every form, since about 98 percent were approved anyway, and check a careful sample.
Grove also says to vary how often you check. Weeks with no problems? Check less.
Problems start? Check more until quality comes back.
Our reading for AI: nobody needs to read every AI draft. Check a careful sample, raise the rate after a miss, and lower it after a clean run.
David Jenyns learned the same lesson in his own agency, as told in SYSTEMology. Some clients paid 120 days late. Automated reminder emails brought only a slight improvement, because clients ignored them.
The fix was a policy. New clients paid up front through automatic billing, and payments started arriving in advance and on time.
Our take: most automation demos are sold to the person who will never run the process. Automate a broken process and you get a faster broken process.
Cut first. Then automate.
Cut 30 to 50 percent of the steps before adding any AI
- 1. Draw every step Leave nothing out to make it look tidy.
- 2. Count them Write the number down. It is your baseline.
- 3. Set a cut target 30 to 50 percent fewer steps in the first round.
- 4. Question each step Keep a step only if something real needs it.
- 5. Then automate What is left is worth making fast.
Steps 1 to 4: work simplification from High Output Management by Andrew Grove. Step 5 is ours.
Automated reminders barely helped; a new payment policy fixed it
| Try 1: automate the chase | Try 2: change the process |
|---|---|
| Three escalating reminder emails | Pay up front through automatic billing |
| Clients mostly ignored them | Less pushback than feared |
| A slight improvement | Payments arrived in advance and on time |
Example from SYSTEMology by David Jenyns, about his own agency.
Change the setup, not the memo
When people keep making the same mistake, stop writing rules. Change the thing they touch.
Michael Gerber tells the best version of this in The E-Myth Revisited. His office had white boards, white walls and blue markers, and the erasers kept streaking ink on the walls.
For more than three weeks they tried meetings, memos, cleaning policies, spot checks and signs that said BE CAREFUL. None of it worked.
Then they fitted a clear plastic collar around each board, four inches wide. The walls stayed clean, and nobody had to think about it.
One collar beat three weeks of memos.
Engineers call this mistake-proofing. You design the step so the error cannot happen.
AI rollouts love the memo. The memo says never paste client data into an AI tool. The collar is a setup that strips client names before anything leaves your systems.
More on that in our guide to AI guardrails that work better than bans.
How to waste an AI budget in four easy steps:
1. Buy seats for everyone before anyone writes down what good work looks like.
2. Point the tool at your messiest process. Call it a pilot.
3. When the output is bad, blame the model and upgrade the plan.
4. Send a memo about responsible use. Then send another one.
That is not an AI strategy. It is a subscription with a hell of a lot of hope attached.
Three weeks of memos failed; a plastic collar fixed it in one move
- Blue ink on white walls The problem Erasers flew off the whiteboards and streaked the walls.
- Meetings, memos, policies, signs Three weeks plus Cleaning teams, spot checks, BE CAREFUL signs. The ink kept coming.
- A four-inch clear collar The fix Fitted around each board. Clean walls, with nobody paying attention.
Story from The E-Myth Revisited by Michael Gerber.
The whiteboard collar has a name: build the step so the error cannot happen
- Mistake-proofing: build the step so the error cannot happen.
- Two names for the same idea as the whiteboard collar.
- The plug only fits one way. No memo needed.
- The machine blocks the mistake, so nobody has to remember.
- A factory method from the 1960s. Older than any AI tool.
Screenshot of Wikipedia, Poka-yoke, taken 2026-10-18
Why do most AI projects fail?
Most AI projects fail for reasons that have little to do with the model. RAND researchers interviewed 65 experienced data scientists and engineers to find out why.
By some estimates, more than 80 percent of AI projects fail. That is about twice the rate of tech projects without AI.
They found five root causes. The first was leaders who misunderstand, or never clearly explain, the problem the AI should solve.
Run the Costume Test on all five, and in our reading four survive the swap. An unclear problem, missing data, chasing new tech over real needs, and weak systems to run it.
Only the fifth is a pure AI problem: a task too hard for AI to do at all.
Boston Consulting Group found only 4 percent of companies creating substantial value from AI. The leaders put people and processes ahead of technology and algorithms.
Same tools. Same models. Different results.
Four of the five root causes survive the Costume Test
| Root cause RAND found | Survives the swap? | Who fixes it |
|---|---|---|
| Leaders misunderstand or miscommunicate the problem | Yes | Leadership and operations |
| Not enough of the right data | Yes | Data owners |
| Chasing new tech over real user problems | Yes | Leadership |
| Weak systems to manage data and run models | Mostly | Operations and IT |
| The problem is too hard for AI | No | A real AI question |
Source: RAND, The Root Causes of Failure for Artificial Intelligence Projects (2024). The swap column is our reading.
Only 4 percent of companies create substantial value from AI
Share of companies at each stage of AI progress
- Not doing much 25
- Still on proofs of concept 49
- Scaling value 22
- Creating substantial value 4
Source: Boston Consulting Group, Where's the Value in AI? (2024).
The people who build AI say most projects fail for reasons that are not the model
- 65 people who build AI were asked why projects fail.
- By some estimates, over 80 percent fail, about twice the rate of non-AI tech projects.
- Cause one: leaders misunderstand or never clearly state the problem.
- Cause two is data. Still an operations job.
- Only this one is a true AI problem.
Screenshot of RAND, The Root Causes of Failure for Artificial Intelligence Projects (RR-A2680-1), taken 2026-10-18
What counts as a real AI problem?
A real AI problem fails the Costume Test. No amount of good management would fix it.
The model states wrong facts with confidence. Answers change from run to run, so you need a new way to check them. Prices shift, so you need to stay free to switch.
Those deserve real technical time. Our guides on how to evaluate AI agents and controlling AI spend cover them.
Everything else goes back to operations, and it needs a name next to it. If nobody knows whose name, start with who should own AI.
One more trap: blaming the person in the seat. Meadows warns that it rarely helps. The next person sees the same information and faces the same pressures.
So change what people can see and what they are measured on. Swapping the person is the weakest move on the board.
Buying more AI is the weakest fix; changing the goal is the strongest
- Change a number A bigger plan, more seats, a newer model, a new hire.
- Change who sees what Put the one number that matters where the decision maker sees it every day.
- Change the rules Who approves, what counts as done, what gets rewarded.
- Change the goal Measure results, like booked calls, not effort, like drafts written.
Adapted from the leverage points in Thinking in Systems by Donella Meadows. The AI examples are ours.
Run the Costume Test with AI: the copy-paste prompt
Paste the prompt below into your preferred AI. It asks for your stuck AI problems one at a time, sorts them, and finds the first one to fix.
It never invents your numbers. It asks you for them and marks any estimate as an estimate.
Bring the list your team complains about. That list is the input.
Paste into your preferred AI. It asks for your team's AI complaints and one workflow, then sorts them and finds the first fix.
Act as an operations analyst. Sort my stuck AI problems with the Costume Test, then pick the first one to fix. Ask me one at a time, with the example: 1. What AI problems or complaints does your team have? List them. (Example: "the AI drafts sound off-brand", "nobody knows who approves AI output") 2. Which workflow do most of them touch? (Example: writing client proposals) 3. What are the steps in that workflow today, in order, and who does each one? (Example: sales call, notes, draft, manager review, send) 4. How long does each step take, and how long does work wait before it? (Example: drafts wait 5 days for review) Rules: - Costume Test: rewrite each problem with "a very fast new hire" in place of "AI". If it still makes sense, label it OPERATIONS and name the old discipline: quality control, ownership, access rules, security, data, process, unit cost or training. If it only makes sense for AI, label it AI. - For each OPERATIONS problem, ask: does the know-how exist but go unused? If yes, the fix is a short written standard people can follow, not a new tool. - Find the slowest step: the one where work waits longest. Fixing any other step first does not count. - Before any automation, list the steps that can be cut. Aim to cut 30 to 50 percent. Question why each step exists. - Prefer a change to the setup that makes the mistake impossible over a new rule or memo. - Never invent numbers. Use only the times and counts I give you. Mark any estimate as an estimate. Answer in this format: Problems: [each one, OPERATIONS or AI, and the discipline] Count: [x of y are operations] Slowest step: [step and wait time] Steps to cut: [list] Mistake-proofing fix: [one change] Where AI fits after that: [one line] Next three actions: 1. 2. 3.
What to do this week
1. Write down every AI complaint your team has. Ten is plenty.
2. Run the prompt and sort them. Count how many survive the swap.
3. Pick one workflow. Find its slowest step, cut the steps nobody needs, and only then add AI.
Then give that workflow one owner and one written standard. Our guide to writing a job spec for an AI agent shows how.
Remember the second list from that room? It was never an AI list.
Put a fast engine in a car with flat tires, and you still go nowhere.
Filip Kovacevic is the founder of Anar Agency in Phoenix. He spent five years running SEO, Meta ads, email, and content for medical practices across the United States, and now does the same work for healthcare, home-service, legal, and professional-service businesses, alongside AI implementation. More from Filip →
Talk it through in 15 minutes
Bring one question from this article. We will tell you what we would do, whether or not you hire us.
Book a 15-Min Chat →