{
  "id": "H09",
  "url": "https://cartforagents.com/guides/ai-visibility-testing",
  "title": "AI visibility testing: a simple before-and-after test for one product fact",
  "description": "Change one product fact, re-ask the same customer questions before and after, and compare against a product you left alone. A one-page test worksheet for Shopify brands.",
  "answer": "Change one fact on one product, and leave a similar product untouched as your comparison. Ask the same list of real customer questions several times before and after the change, under the same conditions, and count how often each product is shown and whether it was the right fit. Call it a win only if the changed product improves on the questions that fact answers while the untouched one stays flat.",
  "take": "Most merchants check once after an edit and see what they hoped to see. Three runs a day for a few days, next to a product you didn't touch, is what separates a real change from noise.",
  "summary": "You fixed a product description. Did it change what AI assistants recommend? One quick check after the edit will not tell you. Here is a simple test any store can run: one change, a fixed list of questions, a product you leave alone for comparison, and a few repeat runs on each side.",
  "claims": [
    {
      "text": "In a supplied-snippet simulation, an invented humidifier with vague copy was picked in 3 of 180 Claude sessions, 0 of 60 on GPT-6.1-Sol, 6 of 60 on Gemini 3.8 Flash and 3 of 54 on Gemini 3.1 Pro. With its facts as a spec line it was picked in 134 of 180, 20 of 60, 19 of 59 and 15 of 55. Each model picked 2 of 8 listings, so random picking is 25%.",
      "source_title": "Cart for Agents: Vague vs specific product facts, simulated picks by Claude, GPT and Gemini models",
      "source_url": "https://cartforagents.com/research/vague-specific-product-facts-simulation",
      "source_date": "2026-10-27",
      "method": "controlled-simulation",
      "limitation": "Models chose from snippets we supplied, with an invented product. Not live ChatGPT or Gemini results, rank or sales."
    },
    {
      "text": "In the same simulation, on GPT-6.1-Sol and two Gemini models, the factual humidifier was picked every time the shopper asked for the facts it stated, and never when the ask was simply a quiet humidifier for a bedroom.",
      "source_title": "Cart for Agents: Vague vs specific product facts, simulated picks by Claude, GPT and Gemini models",
      "source_url": "https://cartforagents.com/research/vague-specific-product-facts-simulation",
      "source_date": "2026-10-27",
      "method": "controlled-simulation",
      "limitation": "One invented product on supplied shelves. Shows why the question list matters, not how live assistants behave."
    },
    {
      "text": "OpenAI says a product appears in ChatGPT's shopping results when ChatGPT perceives it as relevant to the user's intent, considering the query and context such as Memory or custom instructions.",
      "source_title": "OpenAI Help Center: Shopping with ChatGPT Search",
      "source_url": "https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search",
      "source_date": "2026-11-04",
      "method": "documented",
      "limitation": "Living help page. It does not say how much context changes results."
    },
    {
      "text": "Shopify's search preview in Sales channels > Agentic shows the raw outputs of Shopify Catalog search; Shopify says agentic storefronts often re-rank results with their own logic, so the preview is a directional signal, not an exact prediction.",
      "source_title": "Shopify Help Center: Managing agentic storefronts, Reviewing your products' discoverability and ranking",
      "source_url": "https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts/agentic-home",
      "source_date": "2026-11-04",
      "method": "documented",
      "limitation": "Living help page. We have not seen this admin screen ourselves."
    },
    {
      "text": "Shopify reports sales, orders, online store sessions and online store conversion for each AI channel in Sales channels > Agentic, where sales combine referral-based sales and direct checkout. These analytics are not yet available for headless stores.",
      "source_title": "Shopify Help Center: Managing agentic storefronts, Agentic storefronts performance data",
      "source_url": "https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts/agentic-home",
      "source_date": "2026-11-04",
      "method": "documented",
      "limitation": "Living help page. Describes the report, not how many sales any store gets."
    },
    {
      "text": "FoundRate tells its users that AI shopping tools re-check stores every few days to a few weeks, and to give a fix about a week before expecting movement.",
      "source_title": "FoundRate: AI Shopping Visibility for Shopify, FAQ",
      "source_url": "https://foundrate.co/",
      "source_date": "2026-11-04",
      "method": "vendor-claim",
      "limitation": "A vendor's guidance. No assistant documents its refresh timing."
    },
    {
      "text": "For the search \"humidifier for dry winter air large room quiet top fill\", the listing at position 9 said \"low noise level\" in its description, while its Shopify-generated spec list said \"Noise Level: 26 dB\".",
      "source_title": "Cart for Agents: AI visibility audit, A real example",
      "source_url": "https://cartforagents.com/guides/ai-visibility-audit",
      "source_date": "2026-10-26",
      "method": "observed",
      "limitation": "One unpersonalized Shopify Catalog snapshot. Quotes what the record said, not how loud the product is."
    }
  ],
  "sources": [
    {
      "title": "Cart for Agents: Vague vs specific product facts simulation",
      "url": "https://cartforagents.com/research/vague-specific-product-facts-simulation",
      "accessed": "2026-11-04"
    },
    {
      "title": "Cart for Agents: AI visibility audit",
      "url": "https://cartforagents.com/guides/ai-visibility-audit",
      "accessed": "2026-11-04"
    },
    {
      "title": "OpenAI Help Center: Shopping with ChatGPT Search",
      "url": "https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search",
      "accessed": "2026-11-04"
    },
    {
      "title": "Shopify Help Center: Managing agentic storefronts",
      "url": "https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts/agentic-home",
      "accessed": "2026-11-04"
    },
    {
      "title": "FoundRate: AI Shopping Visibility for Shopify",
      "url": "https://foundrate.co/",
      "accessed": "2026-11-04"
    }
  ],
  "limitations": [
    "This is our method. We haven't yet run it end to end on a live store and published the result.",
    "A store-level test like this can show that something changed. It can't prove your edit caused it, because the assistants change too.",
    "The simulation numbers come from models choosing among listings we supplied. Live assistants are noisier and see more products.",
    "Shopify's admin screens and reports change. We read the platform pages on November 4, 2026."
  ],
  "related": [
    "https://cartforagents.com/guides/ai-product-visibility",
    "https://cartforagents.com/guides/ai-shopping-prompt-tracking",
    "https://cartforagents.com/guides/chatgpt-visibility-tracking",
    "https://cartforagents.com/guides/ai-visibility-audit",
    "https://cartforagents.com/guides/ai-visibility-tool",
    "https://cartforagents.com/research/vague-specific-product-facts-simulation",
    "https://cartforagents.com/research/ai-visibility-score-methodology",
    "https://cartforagents.com/guides/sibling-sku-cannibalization"
  ],
  "published": "2026-11-04",
  "updated": "2026-11-04",
  "checked": "2026-11-04",
  "kind": "guide"
}