AI Maturity Assessment: Score Evidence, Then Fix Your Lowest Score

What an AI maturity assessment should actually tell you

The slide says your company is AI Enabled. The ops lead nods along. Then the person who answers customer emails says nobody has checked the AI drafts in a month.

Same company. Two different answers.

In a bad week, only the second answer matters.

Most AI maturity assessments score how a company feels about AI. We score what it can prove.

An AI maturity assessment is a check of how well a company turns AI into dependable, measured work. Most versions hand you a level from one to five and a badge.

Nice badge. Wrong output.

Here is the right one. Score six capabilities from 1 to 3, using only evidence you can show. Your stage is your lowest score, and the fix for the quarter goes there.

Stages still help when you talk to a board. As a plan, they fail.

Maturity ladders measure arrival, and AI work never arrives

Software researchers settled this argument before AI showed up. The authors of Accelerate, Nicole Forsgren, Gene Kim and Jez Humble, studied over 23,000 survey responses from software teams. Their advice: drop maturity models and score capabilities instead.

They gave four reasons. A maturity model has a finish line, so teams stop improving once they reach it. It also puts every team on the same steps in the same order.

It counts tools installed instead of results, which the authors call vanity metrics. And it stands still while the bar moves, so a score that looked strong one year falls short the next.

Their data covered software delivery, not AI. Our reading: AI adoption has all four flaws. The last one bites harder, because the tools change every few months.

There is a second warning in the book. A Forrester report the authors cite found that companies often overestimate their progress, and executives overestimate it most.

So who should score you? The people closest to the work.

DORA, the research group behind the book, keeps a free catalog of capabilities. It now lists AI ones too, like a clear, well-communicated stance on AI use.

Feeling ready is not the same as being ready

Teams overrate their AI skill for the same reason students overrate their test scores. You feel like you know it long before you actually do.

The authors of Make It Stick retell a well-known set of studies by David Dunning and Justin Kruger. Students who scored around the 12th percentile on a logic test believed they ranked near the 68th.

Then it got worse. Shown the answers of stronger students, the weakest group raised their own estimates.

What fixed it? Ten minutes of training in the actual skill.

After that training, their self-ratings got much more accurate.

The lesson for an assessment is simple. A great demo does not make people better judges of their own skill. Practice with feedback does.

The book also offers a better test. Recall that feels easy right after a lesson proves little, while recall that still comes easily after a delay is a good sign.

So we score skills on a real task, done without help, a week after the training. A survey asks how people feel. A task shows what they can do.

Keep the three stages as labels for the board

The three stages are still useful words. They give a board a quick shared picture.

AI Literate: people use AI tools on their own, often as shadow AI, and the value leaves when they do. AI Enabled: AI sits inside named workflows with owners, rules and checks. AI First: new work is designed around what AI can do, and people direct and review it.

Almost nobody is fully at AI First. Treat it as a direction, not a destination.

Use the stages to report. Use the six scores to decide.

Score six capabilities on evidence you can show

A capability is something the company can do on demand, with proof. We score six of them from 1 to 3.

They are workflows, policy, evaluation, spend visibility, ownership and skills. The table below shows what a 1, a 2 and a 3 look like, and what evidence to bring.

The evidence rule matters more than the table. A claim with nothing to show scores a 1.

No proof, no points.

The Accelerate researchers used the same trick in their surveys. They never asked teams whether they did continuous integration, because almost everyone claims it. They asked whether automated tests start every time code is checked in.

Ask about behavior, not labels. Skip "do we evaluate AI output?" and ask for last week's review log.

Policy works the same way. A memo nobody has read scores a 1. The one-page rules in our AI governance framework score a 2 once people have read them.

Score each workflow, not the whole company

Maturity belongs to a task, not to a company. One team can be ready while the team next door is not.

Andy Grove made the same point about people in High Output Management. He called it task-relevant maturity: a person can be highly mature in one job and brand new in another.

His example was a top sales manager moved to run a factory unit. The man did not change, but his results fell until he learned the new work.

Same guy. New job. Rookie again.

Our reading: AI works the same way. Support replies can earn a 3 on evaluation while invoice coding sits at 1.

So give each workflow a rung. AI Engineering by Chip Huyen describes a three-rung ladder Microsoft proposed, called Crawl, Walk, Run.

At Crawl, a person approves every output. At Walk, AI works directly with your own staff. At Run, AI can deal with customers.

The useful part is the gate. Her example: once staff send 95 percent of AI-suggested replies to simple requests without changes, let customers get AI replies for those requests.

Write the gate down before you start. Otherwise the promotion happens on a good feeling and a deadline.

How do you run an AI maturity assessment in twenty minutes?

Put four people in a room for twenty minutes, have each score alone, then compare. The four are whoever owns operations, technology and compliance, plus one front-line user.

Each person scores the six capabilities for each workflow, alone, in five minutes. Then everyone shows their numbers and their evidence.

The disagreements are the most useful thing you will get. Look hardest at the gap between the ops owner and the front-line user.

The boss sees the slide deck. The front line sees Tuesday.

How to get a flattering AI maturity score in four easy steps:

1. Survey only the executives. They feel great.

2. Count licenses and call it adoption.

3. Average the six scores so the weak one disappears.

4. Show the board your best pilot and call it a strategy.

Licenses are not a strategy. They are a receipt.

Your stage is your lowest score, not your average

Take the lowest agreed score as your stage. The lowest capability is the one that causes the incident or the stall.

For example, picture a company with impressive pilots. It scores 3 on workflows and skills, 2 on ownership and spend, and 1 on policy and evaluation.

The average is 2.0, which reads as AI Enabled. The lowest score says AI Literate. That is a Literate company with expensive hobbies.

Averages make a damn nice slide. Incidents ignore the slide.

Then make one move on the lowest capability, using the table below. For spend, use the caps and weekly report from our token spend method.

One move. One quarter. Then score again, with a higher bar for a 3.

Expect the middle to look worse before it looks better. In one year of the Accelerate data, middle teams broke things more often than the lowest teams. The authors suspect new work was pushing out repair work.

Our reading: your first honest review will find problems that were always there. That is the score working, not failing.

Grow capability one team at a time

Do not roll AI out to every team at once. Start with one team working closely with help, then let the rest use what it built.

That is the pattern Manuel Pais and Matthew Skelton set out in Team Topologies. A team should collaborate closely with no more than one other team at once.

The first team works side by side with the platform group and figures out what works. The next team uses the result as a ready service. Later teams adopt an improved version without the close work.

Our reading for AI: one team earns Walk with heavy help. The second team gets the gate, the prompts and the review log as a kit.

Think of a relay race. One team runs the first leg, then hands the next team the baton.

Somebody has to own that kit. Our guide on who should own AI covers the choice.

People need a plan too. Fear of replacement is rational, so answer it with structure, not slogans.

Publish which roles change and how. Train on work time, not weekends. Make the first automations remove drudgery people can see, not jobs nobody mentions.

How is this different from the Microsoft, AWS and Google AI frameworks?

The cloud vendors publish checklists of what to build; our assessment finds what will break first. You can use both.

Microsoft, AWS and Google Cloud each publish a free AI adoption framework. They are detailed, and each leans toward its own platform.

A checklist with forty items invites you to count what you have. Counting feels like progress. It is not a diagnosis.

The staged idea itself is older than AI. The Capability Maturity Model from software engineering measured a company's process, not its tools.

Security analysts hit the same counting trap. In How to Measure Anything in Cybersecurity Risk, Douglas Hubbard and Richard Seiersen build their maturity model around decisions, not tools.

It starts with one question, what should we invest in, and needs very little data to begin. It then climbs to whether each investment works, whether they work together, and whether you catch what is hidden.

Board or regulator asking? The NIST AI Risk Management Framework and ISO/IEC 42001 give you the formal vocabulary. Neither replaces twenty minutes with four people.

Want us in the room instead? Our consultant versus in-house guide explains how we keep the capability with you.

Run the assessment with AI: the copy-paste prompt

Paste the prompt below into your preferred AI. It asks which workflows use AI, what evidence you have, and where your scorers disagreed.

It never invents a number. Every score needs evidence you name, and every gate needs a rate you give it.

It returns a stage per workflow, the lowest capability, and one move for the quarter. Want a second pair of eyes? Send us the result.

What to do this week

1. Book twenty minutes with the four scorers. Ask each one to bring evidence, not opinions.

2. Run the prompt with their scores. Mark the biggest disagreement.

3. Pick the one move for the lowest capability and name its owner. Put the re-score date in the calendar.

Remember the person who said nobody checks the AI drafts? That sentence was your real score.

If evaluation is your lowest score, read how to evaluate AI agents next.

A maturity level shows how tall the building looks. The lowest score shows which floor is weak.