{
  "id": "F09",
  "url": "https://cartforagents.com/research/ai-visibility-score-methodology",
  "title": "AI visibility score methodology: why we retired our own score",
  "description": "Rebuild the failure of our first gap score using missingness, normalization errors and component contribution checks rather than a replacement grade.",
  "answer": "Before you trust any AI visibility or \"gap\" score, compare the weight each part was given with the share of the score it actually produced, and check how it handles blank fields and units. When we recomputed our own retired score over the 162 searches in our October 26, 2026 Shopify Catalog snapshot, review counts produced 64.7% of all points, word matching 28.0%, seller variety 7.3% and result count 0%, against design weights of 25, 30, 30 and 15 out of 100; reading blank ratings as zero supplied 591 of the review points. We stopped using the score that evening and did not replace it with another grade: we now publish plain counts with named units.",
  "take": "We're skeptical of single-number AI visibility scores for product brands. Ours looked rigorous and was mostly a review count, which is why we now publish what came back and what was missing instead.",
  "summary": "Our first catalog study gave every search a 0 to 100 \"gap score\". On paper it weighed four things evenly enough. In practice it was mostly a count of reviews, padded by blank fields read as zero. This is the audit, the record of why we stopped, and the checks to run on any score before you trust it.",
  "claims": [
    {
      "text": "Re-running the original scoring code unchanged on the 162 saved Shopify Catalog responses reproduced every stored score. Across all 162 searches the score awarded 3,774.4 points: 2,442.2 (64.7%) from review depth, 1,057.2 (28.0%) from word matching, 275.0 (7.3%) from seller variety and 0 (0%) from result count.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "A recomputation of our own retired metric on one unpersonalized snapshot of searches we wrote. It describes how this score behaved, not how any other score behaves."
    },
    {
      "text": "Measured by how much of the score's spread between searches each part accounted for, review depth explained 74.0%, word matching 17.3%, seller variety 8.7% and result count 0%.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "Each part's covariance with the total, divided by the total's variance, across 162 searches. A descriptive split, not a causal model."
    },
    {
      "text": "Two parts barely moved: every search returned the 10 results requested, so the result-count part scored 0 on all 162, and 131 of 162 searches returned 6 or more distinct sellers, so the seller-variety part scored 0 on those.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "Follows from the request limit and thresholds we chose; a different request size would change it."
    },
    {
      "text": "The product-level rating was absent on 440 of 1,620 results, in 121 of 162 searches, and no rating carried a count of zero. The score read every absent rating as zero reviews.",
      "source_title": "Cart for Agents: 162 shopper searches in Shopify Catalog (receipts cited by our guides)",
      "source_url": "https://cartforagents.com/research/agent-shopping-research#receipts-cited-by-our-guides",
      "source_date": "2026-10-26",
      "method": "observed",
      "limitation": "Absent means unknown in this response, not zero reviews and not a missing review program. The search count (121) is from our recount on this page."
    },
    {
      "text": "Treating absent ratings as unknown instead of zero, by taking the median of the ratings present, removed 516.4 review points across 159 searches. The other 3 searches had no product rating at all and had taken the full 25 points each from blanks alone. Together that is 591.4 points, 24.2% of all review points and 15.7% of the whole score.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "A sensitivity check, not a corrected score. It still ignores ratings carried on the first variant and any reviews outside the catalog record."
    },
    {
      "text": "Even with absent ratings treated as unknown, review depth still produced 58.8% of the points on the 159 searches with a known rating, because 105 of them had a median rating count below the score's 200 threshold.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "Shows that fixing the blank handling alone would not have rebalanced the score. The 200 threshold was our arbitrary choice."
    },
    {
      "text": "Changing only how absent ratings were handled moved 13 of the 16 categories to a different position in the score's category ranking, by up to 4 places.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "We report how many categories moved, not either ranking. Both rankings came from the retired score and are not published."
    },
    {
      "text": "For \"toddler snowsuit rated for minus 30\", the score was 53.0: 10 points for seller variety, 25 for review depth and 18 for word matching. Only 4 of 10 results carried a product-level rating, while 9 of 10 carried a rating on the first variant, and the word matcher did not treat \"minus 30\" and \"-30 °C\" as the same value.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "One worked search. The temperature-matching error is ours, not the merchants', and is not evidence about how any assistant reads temperatures."
    },
    {
      "text": "For \"humidifier for dry winter air large room quiet top fill\", the first result matched 8 of the 9 search words the matcher used, including \"quiet\", although no decibel figure appeared anywhere in its record.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "observed",
      "limitation": "Word overlap was computed on the title, description and Shopify-generated metadata, which Shopify marks as Inferred. One result, not a rate."
    },
    {
      "text": "We infer that the score measured how few reviews a shelf's records carried and how many of the shopper's words they repeated, not whether the listings answered the shopper's need, and that it was never checked against any outcome such as picks, clicks or sales.",
      "source_title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "source_url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "source_date": "2026-10-27",
      "method": "inferred",
      "limitation": "Our interpretation of the audit above. Alternative readings are possible, but none would make the score a validated measure."
    }
  ],
  "sources": [
    {
      "title": "AI visibility scores: audit weights and retire an invalid metric (how we checked)",
      "url": "https://cartforagents.com/research/ai-visibility-score-methodology#how-we-checked",
      "accessed": "2026-10-27"
    },
    {
      "title": "Cart for Agents: 162 shopper searches in Shopify Catalog (receipts cited by our guides)",
      "url": "https://cartforagents.com/research/agent-shopping-research#receipts-cited-by-our-guides",
      "accessed": "2026-10-27"
    },
    {
      "title": "Cart for Agents: 162 shopper searches in Shopify Catalog (what we retired, and why)",
      "url": "https://cartforagents.com/research/agent-shopping-research#what-we-retired-and-why",
      "accessed": "2026-10-27"
    }
  ],
  "limitations": [
    "This audits one score we built ourselves, on one unpersonalized Shopify Catalog snapshot from October 26, 2026. It does not audit any vendor's score, and its numbers do not transfer to other metrics.",
    "The 162 searches were written by us in 16 categories we chose. The audit shows how the score behaved on them, not how Shopify categories compare.",
    "The sensitivity check (absent ratings as unknown) is a diagnostic, not a corrected score. It ignores variant-level ratings and reviews held outside the catalog record.",
    "A Shopify Catalog response is not a ChatGPT, Copilot or Gemini answer. Neither the retired score nor anything on this page measures what an assistant showed or recommended.",
    "No category ranking, per-search score table or \"opportunity\" list from the retired score is published here, and none will return under another name.",
    "The checklist on this page is proposed. It is not an industry standard, and passing it would not make a score predict sales."
  ],
  "related": [
    "https://cartforagents.com/research/agent-shopping-research",
    "https://cartforagents.com/research/missing-product-review-data",
    "https://cartforagents.com/research/ai-product-data-unit-normalization",
    "https://cartforagents.com/research/shopify-catalog-vs-chatgpt-results",
    "https://cartforagents.com/research/ai-shopping-seller-concentration"
  ],
  "published": "2026-10-27",
  "updated": "2026-10-27",
  "checked": "2026-10-27",
  "kind": "research"
}