How to Track AI Search Visibility: Count a Mention Rate, Not a Rank

How do you track AI search visibility?

A founder asks an AI assistant to name the best roofer in her city. A competitor gets named. She does not.

She sends a screenshot to her team with three question marks. Next morning, a colleague asks the same question. She is named first.

Which screenshot is true? Neither. Each one is a single roll of the dice, and the assistant rolls again every time someone asks.

AI search visibility is how often AI assistants name or cite your company when buyers ask about what you sell. It is not a rank, because the answer gets rebuilt on every run.

So you track it like any other rate. Freeze a set of buyer questions, run each one many times, and count the share of runs that name you.

Then change one page and count again. That weekly loop is the whole system. Nothing fancy.

It is also a textbook marketing engineer job, as our guide to what a marketing engineer is explains. An agent does the counting. A person makes the call.

Hunting a trick to force your name into the answer? Wrong article.

A single check tells you almost nothing

One AI answer is a sample of one. You would not judge an ad on one click, so why judge an assistant on one run?

SparkToro measured how much the answers move. In their study, volunteers ran 12 prompts through three popular AI tools a combined 2,961 times.

The chance of getting the same list of brands twice was below 1 in 100. The same list in the same order came closer to 1 in 1,000.

Their conclusion is the base of this system. A position in an AI answer means almost nothing. How often you show up across many runs is a fair metric.

That matches an old rule of analytics. In Lean Analytics, Alistair Croll and Benjamin Yoskovitz say a good metric is a rate, compares one period with another, and changes what you do next.

A screenshot fails all three tests. A mention rate passes all three.

Rank is the vanity number here. Rate is the real one.

Step zero: make sure AI crawlers can read your site

No count matters if the engines cannot read your pages. Check access first. It takes about ten minutes.

Most AI companies run two kinds of robot. A search crawler fetches pages to answer questions today. A training crawler collects pages to build future models.

They are separate choices. You can say no to training and yes to search, but blocking a search crawler can drop you from that engine's answers, as the crawler help pages say.

Then check two more layers. Your robots.txt file is the rule sheet crawlers read first. A firewall or bot-blocking rule can still turn a crawler away when robots.txt lets it in.

Crawler names and switches change. They sit in the dated box below, so this section stays true.

Check the door before you count who walks in.

Build the question set from buyer jobs, not categories

The questions decide what your number means. Ask the wrong ones and you measure a market nobody shops in.

Write 25 to 50 unbranded questions, the kind a buyer asks before they know your name. Pull them from sales calls, support tickets and reviews, which is the job of a customer truth system.

Write them around the job, not the category. Clayton Christensen's team makes the point in Competing Against Luck: nobody walks into a shop asking for the yellow fats category. They come with a job to get done.

Buyers type their job into an assistant too. "Best roofing company" is a category. "How do I know if my roof needs replacing or just a repair" is a job.

The same book describes brands that pop into mind the moment a job shows up. Our reading: an AI answer works the same way, and names the company most clearly tied to the job.

Tag each question with buyer value. High means the person asking is close to buying. Low means they are just curious.

Then freeze the list for a quarter. Swap questions every week and you are moving the goalposts. Your trend line means nothing.

Read the mention rate as a range, not a point

Your mention rate is the share of runs that name you or cite your page. Named in 3 of 20 runs is a 15 percent rate.

But 20 runs is a small sample. So we never report the rate alone.

Douglas Hubbard and Richard Seiersen give the reason in How to Measure Anything in Cybersecurity Risk. They argue that a measurement is any observation that cuts your uncertainty, stated as a number. It does not have to remove the doubt.

Their tool for small samples is the beta distribution, a standard way to turn hits and misses into a range. Add one to your hits, add one to your misses, and two spreadsheet formulas give you the 90 percent range.

Their own example: one hit in six gives a range of about 5 to 52 percent. Wide? Yes.

Useless? No. Before any data, the same math says 5 to 95 percent, so six observations cut the range nearly in half.

In a spreadsheet, the low end is =BETA.INV(0.05, 1+hits, 1+misses). The high end swaps in 0.95.

Here is our rule, run on made-up counts. Before your rewrite, a question names you in 0 of 20 runs, a range of 0.2 to 13 percent. After it, the question names you in 6 of 20, a range of 17 to 49 percent.

The two ranges do not overlap, so the move is real enough to act on. At 3 of 20, the new range would run from 7 to 33 percent and overlap the old one.

That is not a win yet. Keep counting.

More runs buy a narrower range. That is the real budget question: how sure do you need to be, and what does each run cost?

The Mention-Rate Loop: change one page a week

The Mention-Rate Loop is a weekly cycle. Ask the fixed questions, log every answer, count each rate as a range, fix one page, and count again.

Ask. Run each question several times on every assistant your buyers use. Thirty questions, four assistants, five runs each is 600 answers a week.

That is a job for an agent, not an intern.

Log. Save every answer and every cited link with the date, the assistant and the run number. Save the full text too, because next quarter you will want it.

Count. Work out the rate and range for each question. Note which competitors were named, since their rate is your benchmark.

Fix. Pick the high-value question with the lowest rate. Rewrite or build the one page that should answer it.

Why only one page? You cannot run a true split test on an AI engine, because you cannot divide its users into two groups.

The authors of Trustworthy Online Controlled Experiments describe a design for that case: the interrupted time series. Measure many times before a change and many times after, then compare what happened with what the before predicted.

They also name its weak spot: anything else that changed at the same time. An engine update can move every question at once.

So we keep the untouched questions as a comparison group. If the fixed question moves and the rest stay flat, the page did it. If everything moves, the engine did.

How to fool yourself about AI visibility in four easy steps. Step one: ask one question once and screenshot it.

Step two: rewrite ten pages in the same week, so nobody knows which one worked. Step three: swap the question list when the numbers look bad.

Step four: report a rank to the board. Congratulations, you built a damn expensive mood ring.

Why changing one page can move an AI answer

AI assistants that search the web find pages first and write second. No engine publishes how it ranks, but retrieval systems in general share a few habits.

Chip Huyen lays them out in AI Engineering. Term-based search, like the classic BM25 method, matches the exact words in a question.

Embedding search matches meaning. An embedding is a list of numbers that stands for what a passage means, so it finds pages that say the same thing in other words.

Each one misses something. Meaning-based search can blur exact terms like product names and codes, which is why the book suggests combining the two.

She also describes contextual retrieval: tag each passage with the questions it answers and a short note on the page it came from. A password help article tagged with "I forgot my password" and "I can't log in" gets found by people who never type "reset".

That is retriever design in general. It is not proof of how any public engine ranks your page.

Google's guide to its AI features pushes the other way on one point. It says you do not need to chop pages into chunks, add special AI files, or rewrite text just for AI.

Who is right? Both, if you write for the person first. Our decision rule: every section answers one question on its own, in the words buyers use, with your exact product names.

There is one controlled test of page edits. In the GEO study, adding quotations, statistics and cited sources raised a page's visibility in a test answer engine, while stuffing in keywords lowered it.

So the fix list is short. Put the answer in the first lines, add your own dated numbers, and source every claim. No tricks required.

Our E-E-A-T checklist covers the trust side. For which kinds of sites get cited at all, read what AI answer engines actually cite.

Who runs it: the agent counts, a person decides

This is a marketing engineering system because most of it is plumbing. An agent runs the questions, saves the answers and fills the table.

A person picks the page and approves the rewrite. The agent never publishes, because a wrong page on your site is a costly mistake.

Brief the agent like a new hire: which questions, which assistants, how many runs, where the table lives, and what it must never do. Our agent job spec shows the format.

The ranges then become one line in the Monday growth cockpit memo. The question, this week's range, last week's range, and the call.

Expect slow numbers. A rewritten page must be crawled again before any engine can use it.

Check where your buyers ask in the first place, too. Our Ask-Ratio analysis of Google vs AI assistants found the split changes a lot by topic.

And grade the system on what buyers did, not on how many answers it logged. That is the rule in how to measure a marketing engineer.

Run your first week with this prompt

The prompt below reads one week of counts. Paste it into your preferred AI with your questions, hits and runs, and it returns rates, ranges and the one page to fix.

This week, write 25 questions in your buyers' own words. Run each one five times on the assistants they use, and log every answer.

Next week, fix one page. The week after, count again.

Rather have the system built inside your company? That is our marketing engineering work. If the page work is the gap, our SEO service handles it.

Remember the founder with two screenshots? She was asking the wrong question.

Not "are we in the answer?" but "how often?"

One screenshot is a single at-bat. A mention rate is the batting average.