Why an "AI rank" is not a real number

Published 2 September 2026 · our method

Ask an AI assistant the same question twice and you get two different answers. Not slightly different wording — different businesses. So a chart showing your "position on ChatGPT" is one dice roll drawn as a fact. The honest number is mention rate: how often you are named across repeated samples of the same question. Here is our method, and the runs that made us insist on it.

The same question, asked twice, back to back

On 31 August 2026 we asked, twice: "Who are the best interior designers in Kharghar, Navi Mumbai? Name specific businesses I should contact." Same engine, same wording, separate calls, one immediately after the other.

Six names held. Three did not. Three businesses were "on ChatGPT" or "not on ChatGPT" depending on which minute you asked. If we had run this once and sold you a position, we would have sold you a coin toss.

The same category, a day apart

We sampled "real estate developer, Navi Mumbai" on 31 August and again on 1 September. Lakhani Builders was named in 6 of 6 answers on the first day and 1 of 5 on the second. The first run named 39 different developers; the second named 21.

We are not going to pretend that is a clean experiment. The two runs used slightly different wording for the city, and one of the second run's six calls failed with an API error, so we report it as five answers rather than six. Both facts weaken it as a before-and-after and neither rescues the idea of a rank. If a number can move that far in a day, across wording you did not choose, it is not a position — it is a sample.

The name itself is not stable

We record every business name exactly as the answer writes it. In our Kharghar interior design run, one firm appeared as DesignCafe in three answers and Design Cafe in the other three — never both in the same answer, so it was actually named in all six. In Vashi, the same firm arrived as DELECON, DELECON Design Co. and Delecon Design Company across six answers.

A ranking tool has to silently decide those are one competitor or three, and you never see which it chose. We publish the raw rows and tell you what we think they add up to. You can disagree with our arithmetic; you cannot disagree with a hidden merge.

So we report mention rate

Mention rate is the share of repeated answers that name the business. Six of six is visibility. One of six is noise. That is it — no position, no score out of a hundred, no index.

It is a blunt number and we prefer it for exactly that reason. It survives the non-determinism, it is arithmetic anyone can redo from the raw answers, and it cannot be dressed up to look like progress.

How we sample

• Real questions. Three per area, phrased the way customers phrase them — "Who are the best…", "I need a … for my 2BHK in …", "Recommend a good, reasonably priced …" — with the trade and the node substituted in.
• Repeats. Each question asked at least twice in separate calls. Three questions, two repeats, six answers per area.
• One engine at a time. Today that is Claude with live web search. We report per engine and never average across engines, because they do not agree and an average would hide that.
• Names extracted from the answer. We ask the assistant to end with a machine-readable list of the businesses it named, and fall back to reading its own numbered headings if that line is missing.
• Counted per area, per run, per name-as-written. No merging, no weighting, no smoothing.
• Dated. Every count belongs to a date. A mention rate without a date is a claim, not a measurement.

What we do with a run that goes wrong

Our first pass at Kharghar, at 12:32 UTC on 31 August, recorded zero businesses across three answers. The answers were not empty — they named firms in prose. Our extractor simply failed to catch them. We threw that pass away and re-ran with the extraction fixed; the replacement pass finished five minutes later, and the re-run is the data on the Navi Mumbai interior design page.

The failed call in the 1 September developer run is on the record for the same reason. A measurement you only publish when it flatters you is advertising.

What six samples can and cannot tell you

Six answers is a small sample, and we say so on every page that uses one. It separates "named nearly every time" from "named once", which is the distinction that matters to a business owner. It cannot support fine comparisons: 3 of 6 against 4 of 6 is not a finding, and we will not report it as one. Nor does a count from 31 August tell you what an assistant says today.

The tell, when you are shopping for this

Ask any provider four questions. Which engine did you ask? How many times? On what dates? Can I see the answers? If the reply is a dashboard with a rank on it and no answer to those four, the number on screen was not measured.

Our version of the answer is a weekly receipt: the exact prompts, whether you were named, every change we made with before-and-after copies, a re-check proving the change is live, and the four-week delta. What we will not give you, at any price, is a rank. More on what this work is called and how it is measured, or see what it costs.

Want to know what the AIs say about your business? WhatsApp +91 72084 61733 — the first look is free.

← all guides