On a Monday call, the head of marketing shares a screenshot. She asked ChatGPT for the best invoicing tools for small agencies, and it listed five products. Theirs was not among them; two competitors were. By Wednesday there is a task in the backlog called "fix our AI visibility", and nobody can say what the current number is, because the only data point is that one screenshot.
Run the same prompt again and the list may change. Ask from a different account, a different country or with web search switched off, and it changes again. That variability is the first thing any measurement has to absorb.
After tearing down several setups teams use for this, from manual spreadsheets to commercial trackers, the parts that matter reduce to four: the prompts you ask, how many times and where you ask them, what you count in each answer, and how much error your numbers carry. Everything else is presentation.
Thing one: the prompt set decides what you are measuring
A prompt set is your sample of the questions real buyers ask. If it is wrong, every metric built on it is precise and meaningless.
Most first attempts fail here in one of two ways. Some teams write prompts that contain the brand name ("Is Acme a good invoicing tool?"), which measures sentiment, not visibility, because the model will always mention a brand you named. Others write 10 generic prompts that no buyer types, such as "top software companies", and conclude they are invisible.
A useful set mirrors the buying journey. Group prompts into 3 or 4 intents: category discovery ("best invoicing software for agencies of 10 people"), comparison ("Acme vs Harvest for agency billing"), problem led ("how do I stop chasing late client payments") and alternative seeking ("cheaper alternative to FreshBooks"). Write 20 to 40 prompts per intent, phrased the way customers phrase them. Sales call recordings, support tickets, Google Search Console queries and the "People also ask" boxes are good sources.
Keep two lists separate: unbranded prompts measure whether you appear at all, and branded prompts measure what the model says about you. Mixing them inflates the mention rate. Freeze the set once it is built, version it, and change it on a schedule, perhaps every quarter, so a trend line compares like with like.
Thing two: sample repeatedly, across assistants and contexts
LLM answers are not deterministic. The same prompt sent to the same model can produce different brand lists on consecutive runs, especially for categories with many comparable vendors. One run per prompt gives you a coin toss dressed up as data.
Run each prompt several times, at least 5, and more for prompts that matter commercially. Spread runs across the assistants your buyers use. For most B2B and consumer categories that means ChatGPT, Google's AI Overviews and AI Mode, Gemini, Perplexity, Microsoft Copilot and Claude, with weights based on what you know about your audience.
Context changes answers too, so decide it on purpose:
- Web search on or off. Assistants with live search draw on current pages and cite them; without search, they rely on training data that can be a year or more old.
- API or consumer app. API responses usually skip the personalization, memory and location signals that the consumer apps apply.
- Location and language. A query from Lisbon and one from Chicago can return different vendors, particularly for local services.
- Logged in or fresh. Memory features in consumer apps mean your team's own accounts are biased samples. Use clean accounts or the API.
Commercial trackers such as Profound, Peec AI, Otterly.AI or Ahrefs Brand Radar automate this sampling at scale. Whatever tool you use, check how many runs it makes per prompt and in which context, because a vendor that runs each prompt once per week is reporting noise with a nice chart.
Thing three: four metrics worth counting in each answer
Once answers come back, parse each one for a small set of fields. Teams that track 15 metrics tend to act on none.
Mention rate: the share of answers that name your brand at least once. It is the headline number for unbranded prompts.
Share of voice: your mentions divided by all brand mentions in the category set, across all answers. It shows whether you are gaining ground on competitors or simply riding a category that assistants mention more often.
Position: where you appear when you appear. First in a list of 5, or fifth? A rank weighted score, for instance 1 point for first place down to 0.2 for fifth, separates "mentioned in passing" from "recommended".
Citation rate: for assistants that show sources, how often your own pages, or pages about you, are among them. This is the metric content and SEO teams can most directly influence, because it ties answers back to specific URLs.
Sentiment and factual accuracy deserve a check too, mostly on branded prompts. An assistant that mentions you often but states an outdated price or a discontinued plan is a problem the mention rate will not show. Log these as issues with the exact answer text rather than trying to score them on a scale.
mention_rate = answers_with_brand / total_answers
share_of_voice = brand_mentions / all_tracked_brand_mentions
rank_score = average over answers of (1 / position), 0 if absent
The formulas are simple on purpose. Extraction is where errors creep in: product names with common words, abbreviations and misspellings all need rules. A brand called "Notion" or "Linear" has to be matched carefully, or every sentence about notions and linear regression counts as a mention.
Thing four: read the error bars before the trend
This is the part most dashboards hide. A mention rate measured on a few hundred answers moves around by chance, and teams react to movements that are pure noise.
Treat each answer as a yes or no trial. With n answers and an observed mention rate p, a rough 95 percent margin of error is 1.96 times the square root of p(1 minus p) divided by n. At 50 answers and a 30 percent rate, the standard error is about 0.065, so the margin is close to plus or minus 13 points: anything from 17 to 43 percent is consistent with the data. At 500 answers, the margin shrinks to roughly 4 points.
That arithmetic has practical consequences. A report that says "mention rate rose from 28 to 34 percent this week" on 60 answers per week has told you nothing. Compare monthly figures on larger samples, or show the interval next to each number. Changes in model versions also break trend lines: when an assistant ships a new default model, mark the date and expect a step change that has nothing to do with your marketing.
The strongest objection to all of this is cost. Running 120 prompts, 5 times each, across 5 assistants is 3,000 answers per cycle, and API costs plus analyst time add up. For a small company, a lighter approach is reasonable: 30 key prompts, 3 runs, 2 or 3 assistants, checked monthly. It will miss small shifts, but it will catch large ones, which are the only ones a small team could act on anyway.
A worked teardown: one quarter of tracking for a SaaS brand
The following mini case is illustrative, a composite rather than a specific company. A project management tool for creative agencies, competing with 6 named rivals, built a tracking setup in one quarter.
The team wrote 120 unbranded prompts across 4 intents and 20 branded prompts. They ran each unbranded prompt 5 times on 4 assistants through APIs with web search enabled, giving 2,400 answers per monthly cycle. Their first cycle showed a mention rate of 18 percent against 41 percent for the category leader, with a citation rate of only 6 percent. Problem led prompts were worst: 9 percent.
Reading the citations explained much of it. The leader appeared in third party comparison articles and review sites that assistants cited repeatedly, while the team's own pages answered few of the problem led questions directly. Over the quarter they published 8 pages answering specific questions, such as how to track agency retainer hours, and pursued inclusion in 3 comparison roundups.
By the third cycle, the mention rate on problem led prompts had risen to 17 percent. With about 600 answers in that intent per cycle, the margin was roughly 3 points, so the change was real, not noise.
The overall mention rate rose less, from 18 to 22 percent, and the team could not separate their work from a model update that happened mid quarter. They recorded that limitation in the report instead of claiming the full gain.
Running it yourself: the procedure and a trust checklist
The setup above is reproducible with a spreadsheet, a short script against the model APIs and a few days of analyst time for the first cycle. Later cycles take a fraction of that, because the prompt set, matching rules and report template already exist. Budget the first cycle at roughly 3 to 5 working days, most of it spent on prompts and on validating brand matching.
Here is the procedure the team followed, in order:
- Collect 200 to 300 candidate prompts from sales calls, support tickets and search data.
- Cut to 120 unbranded and 20 branded, grouped by intent, and version the list.
- Choose assistants and context: API, web search on, one market and language.
- Run each prompt 5 times per assistant and store full answer text with timestamps and model versions.
- Extract brand mentions with matching rules tested on 100 hand labelled answers.
- Compute mention rate, share of voice, rank score and citation rate per intent and assistant.
- Report each metric with its margin of error and flag model changes on the timeline.
- Pick 2 or 3 actions per quarter based on which intent and which cited sources show the biggest gap.
Before trusting any LLM visibility report, your own or a vendor's, check it against this list:
- The prompt set is written in buyers' words and excludes the brand from unbranded prompts.
- Each prompt runs at least 5 times per assistant per cycle.
- Context is fixed and documented: search setting, API or app, market, language.
- Raw answers are stored, so any number can be traced back to text.
- Brand matching was validated on hand labelled samples.
- Every metric shows a sample size or a margin of error.
- Model version changes are marked on trend charts.
The question nobody has settled yet is what a mention is worth. A buyer who reads your name in an answer and never clicks leaves no trace in analytics, and assistants with memory will soon give each user a different answer to the same prompt. If the answer stops being a shared object, what exactly will a mention rate be measuring?