A product-URL checker can prove what it fetched, which fields it read, which model or route it asked, with what prompt and context, and when. It cannot prove what an assistant recommends to your customers. Their answers depend on their own questions and context, and the checker has neither.
So before you act on a result, test the checker. Its report should show the record it received and the sources it used, say which field each finding came from, treat missing data and failed fetches as unknown rather than zero, and keep its conclusion inside the sample it actually ran. The acceptance tests below make each of those checkable in a few minutes.
Three kinds of checker, three kinds of proof
“AI visibility checker” covers tools that do very different things. Here are three, described from their own method pages.
| Kind | Example, as the vendor describes it | What it can prove | What it can’t |
|---|---|---|---|
| Readiness, from your site’s code | AgentReady reads public documents (homepage, product pages, robots.txt, sitemap, llms.txt) and scores 23 always-on checks plus 6 conditional ones | Which signals your public files contain, on the day it read them | Whether any assistant reads, trusts or picks you |
| Live prompts to a model | eCommerce Insights runs two buyer-style prompts through GPT-4o with web search. usegeo asks three product questions, each twice | What that model said to those prompts, at that time, under the tool’s context | What ChatGPT, Gemini or Perplexity tells a real shopper with their own history |
| Catalog or record reads | ARC reads structured data and runs browser tests to add-to-cart. Our own research reads Shopify Catalog responses | What a specific record or route returned, field by field | How an assistant ranks or describes the product |
All three are useful. The mistake is reading one as another: a readiness grade as a visibility result, or two prompts as “ChatGPT doesn’t recommend you.”
Why a live-prompt check isn’t the shopper’s answer
OpenAI says ChatGPT picks shopping results by relevance to the shopper’s query and context, such as Memory or Custom instructions. A checker calling a model through an API has none of that context. It may also not be the same product surface: a model with web search, called by a vendor, is not the ChatGPT app. usegeo says as much on its own page: three questions are a small sample, answers change over time, and assistants answer differently for different people. That’s an honest limitation, and every live-prompt checker shares it.
If you want to see what ChatGPT returns for yourself, run controlled manual checks. Our protocol for manual ChatGPT checks covers how. Even that shows your sessions, not anyone else’s.
The acceptance test pack
These are our proposed tests. Each one names what to feed the checker or look for in its report, and what a pass looks like. You don’t need every test for every tool: a readiness checker doesn’t run prompts, so the context tests don’t apply. Mark those “n/a”, not “pass”.
Provenance: can it show its work?
| Test | What to look for | Pass | Fail |
|---|---|---|---|
| P1. Received record | The exact URL fetched, the date and time, and what came back | The report shows or links the fetched content, or lists the documents read | A score with no record of what was read |
| P2. Field attribution | Each finding names the field it came from: title, description, structured data, generated metadata, reviews | “Decibel figure found in the description” | “Noise: covered” with no field |
| P3. Source attribution, live checks | Which pages the model cited, and whether yours was one | Cited sources listed per answer | “You were mentioned” with no answer text |
| P4. Reproducibility | Enough detail to rerun it: the method, weights or prompts | Published method, or the prompts shown in the report | “Proprietary AI analysis” |
P2 matters more than it looks. In our October 26, 2026 Shopify Catalog snapshot for “humidifier for dry winter air large room quiet top fill”, a decibel figure was in the description of 4 of the 10 listings (positions 2, 3, 5 and 7). Counting Shopify-generated metadata too, it was in 6 of 10 (adding positions 6 and 9). No title had one. A checker that says “4 of 10” and one that says “6 of 10” are both right, but only if they tell you which field they read.
Context: does it say what it asked, and of what?
| Test | What to look for | Pass | Fail |
|---|---|---|---|
| C1. Model and route | The model name and version, and whether it’s an API, an app or a catalog | “GPT-4o with web search, via API” | “ChatGPT” with no detail |
| C2. Prompt text | The exact prompts | Shown verbatim | Hidden, or described as “buyer questions” |
| C3. Buyer context | Signed in or out, location, any memory or instructions | Stated, even if it’s “none” | Not mentioned |
| C4. Runs and date | How many times each prompt ran, and when | “Each asked twice on [date]” | One run, undated |
Missing inputs and errors: does a blank stay blank?
This is where checkers most often go wrong, ours included. Feed each test a product you know.
| Test | What to feed it | Pass | Fail |
|---|---|---|---|
| M1. Missing field | A product with no rating, or no stated size | Reported as unknown or not present | Scored as zero, or as “poor reviews” |
| M2. Failed fetch | A URL that’s blocked, redirected or offline | An error state with no score | A low score, as if the page were empty |
| M3. Condition not met | A product with no structured data at all | Checks that need it are skipped and say so | Every dependent check fails separately, compounding the penalty |
| M4. Field the tool can’t see | A fact that’s only in an image or a PDF | “Not readable by this tool” | “Missing”, as if you never published it |
Some vendors get this right in public. AgentReady skips a conditional check when its precondition is absent: no Product schema means no GTIN check. Its proof page shows a property it couldn’t scan as having no score, rather than inventing one. ARC keeps inferred scores out of its rates and says a missing score is not a zero. Note the difference, though: AgentReady scores an absent signal as a fail, which is a design choice, not an error. If you’re testing for M1, ask whether “absent” means “you didn’t publish it” or “we couldn’t find it”. Those need different fixes.
Our own retired score failed M1 badly. In our 162-search snapshot, the product rating was absent on 440 of 1,620 result occurrences, and none of the 1,180 that were present was an explicit zero. Of the 440, 106 carried a rating on the first variant. Our score read every absent rating as zero reviews. The audit shows what that did to the result, and the missing-review research covers why absent never means zero.
Unsupported conclusions: does the verdict fit the evidence?
| Test | What to feed or check | Pass | Fail |
|---|---|---|---|
| U1. Equivalent units | A product stating “-30 °C” against a check for “minus 30” | Recognized as the same value | Reported as missing |
| U2. Word versus fact | A listing that says “quiet” with no decibel figure | “Quiet: words only, no value” | “Quiet: covered” |
| U3. Sample size | The number of answers behind a verdict | The verdict is scoped: “0 of 6 answers on one model, on one day” | “ChatGPT doesn’t recommend you” |
| U4. Readiness versus visibility | A readiness grade | Called a readiness grade | Called visibility, ranking or “how AI sees you” |
| U5. Repeatability | Run the same check twice, a day apart | Differences are shown and explained | A different score with no explanation |
U1 and U2 are both mistakes our own retired score made. It treated “minus 30” and “-30 °C” as different values, and it gave the first humidifier result credit for “quiet” when no decibel figure appeared anywhere in its record.
How much can two or six answers tell you?
Live-prompt checkers run a handful of answers because each one costs money. That’s reasonable. The problem is a verdict that sounds bigger than its sample.
| Result | 95% Wilson interval for the true rate | What you can honestly say |
|---|---|---|
| 0 of 2 answers | 0% to 65.8% | Nothing useful about how often you’d appear |
| 0 of 6 answers | 0% to 39.0% | You didn’t appear in these six. You might still appear in a third of similar answers |
| 2 of 6 answers | 9.7% to 70.0% | You appeared sometimes. The rate is unknown |
These intervals assume each answer is independent, and repeated prompts to one model aren’t, so the real uncertainty is wider. The scoring questions that sit on top of samples like these (denominators, weights, missing data) are in our guide to interrogating AI visibility scores.
A worked check: three public method pages
We read each tool’s public method page on October 27, 2026 and marked what the page documents. We didn’t run the tools. “Not documented” means the page we read doesn’t say, not that the tool fails.
| Test | AgentReady (readiness) | eCommerce Insights (live prompts) | usegeo (live prompts) |
|---|---|---|---|
| P1. Received record | Lists the documents read | Says it reads the product page; both full responses with every citation come in the emailed report | Shows the answers, brands named and sites cited |
| P4. Method published | Weights and a worked example published | Steps described | Scoring rule described |
| C1. Model and route | n/a | GPT-4o with web search | ChatGPT, Gemini and Perplexity; model versions not documented on the page we read |
| C2. Prompt text | n/a | Described as a category-search and a buying-intent prompt; whether the prompts are shown verbatim is not documented on the page we read | Its sample report lists “the 3 questions we asked” |
| C4. Runs | n/a | Two prompts | Three questions, each asked twice on each assistant |
| M2. Failed fetch | Unscannable property shown with no score, on its proof page | Not documented on the page we read | Not documented on the page we read |
| U3. Sample scoped | n/a | Says one engine is a sample | Says three questions are a small sample |
None of these pages claims to show what assistants tell real shoppers, and two say plainly that their sample is small. That’s the bar.
How we checked
- Vendor pages. On October 27, 2026 we read the public method pages of AgentReady (homepage, how-it-works and proof pages), eCommerce Insights’ ChatGPT Product Visibility Checker, usegeo’s AI Visibility Checker and ARC’s product-data page. We quote or paraphrase only what those pages say. We didn’t run any checker.
- Snapshot figures. From our saved Shopify Catalog responses of October 26, 2026: the humidifier search counts and the 162-search rating counts are the figures published in our receipts.
- Intervals. 95% Wilson score intervals, computed by script for 0 of 2, 0 of 6 and 2 of 6.
- What we didn’t do. We didn’t compare checker outputs against each other, against live assistant results or against sales.
Your next step
Run the checker you use on two products you know well: one with a fact you know is missing (no rating, no size) and one URL that’s offline. Then score the report against tests P2, M1, M2 and U3. If it scores the missing fact as zero or gives the offline URL a grade, don’t act on its other findings until the vendor explains how it handles blanks.