Before you trust an AI visibility score, ask five things:
- What was sampled? Which products, which questions, which assistants, which dates.
- What’s the denominator? Answers, questions, products or stores.
- What happened to missing data? Counted as a fail, a zero, unknown, or left out.
- How are the parts weighted, and what do they actually contribute?
- Was it ever checked against an outcome, such as clicks or sales?
A score that can’t answer those describes its own method, not your visibility. We know because our own first score failed these checks, and we retired it. Below are five cases worked from public score methods, then the checklist.
Every score is a fraction
“You scored 41.” “87% of stores never came up.” “Grade B.” Each of these is something divided by something. The top of the fraction gets the marketing. The bottom decides what the number means.
| Denominator | Example statement | What it actually measures | What to ask |
|---|---|---|---|
| Answers | “Named in 4 of 6 answers” | How often one product or brand appeared in the answers the tool collected | How many answers, to which questions, on which assistants, when |
| Questions | “Visible for 30% of prompts” | Whether it appeared at least once per question | How many runs per question, and what counts as “at least once” |
| Products | “12 of 40 products found” | Coverage across a catalog | Which products were tested and how they were chosen |
| Stores | “87% of stores never came up” | Share of stores with zero appearances | How many questions each store got |
| Checks | “23.5 of 31” | Share of a weighted checklist passed | Which checks, what weights, what counts as a fail |
Two scores with different denominators can’t be compared, even if both are out of 100.
Case 1: “never came up once” depends on how many times you asked
FoundRate’s scan page says that in its 82-store study, 87% of stores never came up once. That’s about 71 stores. It’s a striking number, and it’s a vendor statistic. The page we read doesn’t say how many questions each store got, on which assistants or when.
That matters more than it sounds, because “never once” gets easier to hit the fewer times you ask. Here’s the arithmetic for an invented product with a steady 10% chance of appearing in any one answer:
| Answers collected | Chance it never appears |
|---|---|
| 3 | 72.9% |
| 6 | 53.1% |
| 10 | 34.9% |
| 20 | 12.2% |
The product is identical in every row. Only the sample size changes. Real answers aren’t independent and real rates aren’t constant, so treat this as a picture of the problem, not a model of any tool.
Ask: how many answers per product or store sit behind the zero? The same question applies to any “not found” verdict. If the answer is a handful, the verdict is “not seen in this sample”, nothing stronger.
Case 2: a blended score can rank a store that appears everywhere last
usegeo describes its score clearly. It counts how often a brand is named and where: full points in an answer’s top three, two-thirds lower down, one-third when an answer only links to the website, averaged across assistants. It asks three questions, each twice per assistant.
That’s a reasonable design. It also blends how often and where into one number, and blends can reorder things. Here’s a rule of that shape applied to three hypothetical stores, six answers each, on one assistant:
| Hypothetical store | What happened in 6 answers | Score under the rule |
|---|---|---|
| A | Named in the top three in 3 answers, absent in 3 | 50.0 |
| B | Linked in all 6 answers, never named in the text | 33.3 |
| C | Named lower down in 5 answers, absent in 1 | 55.6 |
Store B appeared in every answer and scores lowest. Store A was missing from half and beats it. Neither ordering is wrong, but they answer different questions, and the single number doesn’t tell you which question you’re looking at.
Ask: can you see the parts? “Named in X of Y, linked in Z of Y” tells you what to fix. A blend alone doesn’t. This is our arithmetic on invented answers, not usegeo’s computation.
Case 3: missing data, a design choice that moves the number
AgentReady publishes its whole method, with weights and a worked example, which is exactly why it makes a good case. Its score is a weighted average of 23 always-on checks totalling 31 base weight. A pass earns full credit, a warn half, a fail none. An absent signal counts as a fail. Its worked example, a stand-in store, earns 23.5 of 31, shown as 76, grade B.
Two of the failed checks in that example are an llms.txt file and an agents.md file, each with a weight of 1. Here’s what happens to the same store under other reasonable choices:
| How the two missing files are treated | Calculation | Score |
|---|---|---|
| Fail, as published | 23.5 ÷ 31 | 76 |
| Warn (half credit) | 24.5 ÷ 31 | 79 |
| Left out as not applicable | 23.5 ÷ 29 | 81 |
Nothing about the store changed. A five-point swing came from one decision about what absence means. AgentReady’s own proof page says independent studies have found no measurable ranking or citation lift from an llms.txt file, and that it treats the file as hygiene, not a ranking lever. That’s a defensible position, and publishing the weights lets you see its effect. Most scores don’t let you check this at all.
Our own retired score made the harsher version of this choice. In our 162-search Shopify Catalog snapshot, product ratings were present on 1,180 of 1,620 result occurrences and absent on 440, and none was an explicit zero. Our score read every absent rating as zero reviews. That supplied 591.4 of its review points. The full audit has the numbers, and the missing-review research explains why absent never means zero.
Ask: when the tool can’t find something, does it score a fail, a zero, a warn, or leave it out? Then ask how much your score would move under each.
Case 4: weights on paper versus what decides the ranking
A score’s stated weights tell you what its designers intended. They don’t tell you what drives the result. A part that almost never varies can’t affect a ranking however heavily it’s weighted, and a part that varies a lot can dominate a ranking it was meant to be a quarter of.
That’s what happened to us. Our retired score was designed as 25% review depth. Recomputed over 162 searches, review depth produced 64.7% of all the points it awarded and explained most of the difference between searches. A score built from four parts was mostly a review count.
Caps are a related case. AgentReady caps the score at 49 when robots.txt blocks an AI search crawler, so one check can override all the others. That’s stated openly and has a clear reason: data an engine can’t read doesn’t help. It also means two stores at 49 might be nothing alike underneath.
Ask: for each part, what share of the points did it actually produce across the products scored? Is there any cap or override, and how often does it fire?
Case 5: time, and whether the score was ever checked against anything
Live answers change. usegeo says so on its page, and asks every question twice because runs differ. A score without a date and a run count is a single photograph presented as a trend.
Good labelling looks like ARC’s product-data page. Each figure is marked measured, inferred or not measured. Inferred scores are kept out of its rates. Its snapshot carries a version and date, 12 October 2026, and one table is marked as live saved checks, not a frozen snapshot.
The last question is the one most scores can’t answer: has anyone checked it against an outcome? Not “does the number go up when you fix things” (the fixes are what it measures, so of course it does), but does a higher score go with more assistant referrals, picks or sales? If nobody has checked, the score is a description. A useful description, possibly, but not a prediction.
The score interrogation checklist
Our proposed checklist. Send it to the vendor, or run it yourself if the method is published. A good tool won’t pass every row, but it should be able to answer every one.
| # | Question | A good answer sounds like | A red flag sounds like |
|---|---|---|---|
| 1 | What exactly was sampled? | “These 5 products, these 3 questions, ChatGPT and Gemini, on October 20” | “Your store’s AI presence” |
| 2 | What’s the denominator? | “Named in 4 of 6 answers” | A percentage with no count |
| 3 | How many runs per question? | “Each asked twice; both shown” | One run, or not stated |
| 4 | What buyer context was used? | “Signed out, US, no memory” | Not mentioned |
| 5 | How is missing data scored? | “Absent is a fail; here’s the list” or “Unknown, excluded” | “Missing ratings count as zero” |
| 6 | What do the weights contribute in practice? | Per-part points for your result | Weights only, or “proprietary” |
| 7 | Are there caps or overrides? | Stated, with the rule | Discovered when your score won’t move |
| 8 | When was it measured, and will it change? | A date and a version | “Live” with no date |
| 9 | Can I reproduce it? | Published method, or the raw answers | No method, no answers |
| 10 | Has it been checked against an outcome? | “Not yet; it’s a description” is an honest answer | “Proven to increase sales”, with no study |
Rows 2, 5 and 10 are the fast ones. If a vendor can’t give you a count, a missing-data rule and an honest answer on outcomes, you’ve learned what you need about the score.
For a single product-URL check, the questions shift to what the report can prove about one record: what it fetched, which fields it read, how it handles errors. That’s our checker acceptance test pack.
What to use instead of one number
You don’t have to give up measurement. You give up the blend. Keep counts with their denominators, field by field and outcome by outcome:
| Instead of | Keep |
|---|---|
| “Visibility score: 41” | “Named in 4 of 6 ChatGPT answers, 2 of 6 Gemini, on October 20, signed out, US” |
| “Readiness: B” | The list of failed checks, each with its fix |
| “Data quality: 72%” | Which facts are stated, in which field, and which are unknown |
| “AI traffic is up” | Sessions from the AI channel, with the date range and the channel’s definition |
A SKU dashboard shows one way to keep those apart without collapsing them into a grade.
How we checked
- Vendor methods. On October 27, 2026 we read the public method pages of AgentReady (how-it-works and proof pages), usegeo’s AI Visibility Checker, FoundRate’s scan page and ARC’s product-data page. We quote only what those pages say and ran none of the tools.
- Recomputation. We recomputed AgentReady’s published worked example from its table: 23.5 earned of 31 base weight. Removing its two weight-1 fails (llms.txt and agents.md) gives 23.5 of 29; scoring them as warns adds 1 point, giving 24.5 of 31. Its published grade bands put 76, 79 and 81 all in grade B.
- Arithmetic. The never-appears table is (1 − 0.1)ⁿ for n answers. The blended-score table applies full, two-thirds and one-third credit to six hypothetical answers per store. Both were computed by script.
- Our retired score. Figures come from our published audit of the 162-search Shopify Catalog snapshot from October 26, 2026.
- What we didn’t do. We didn’t run any score, validate any score against outcomes, or compare vendors on the same products.
Your next step
Take the last visibility score anyone gave you and send rows 2, 5 and 10 of the checklist to whoever produced it: what’s the denominator, how is missing data scored, and has it ever been checked against an outcome. Write the answers next to the score before you quote it again.