A shopping agent doesn’t run one ranking. Your product has to clear five stages: it must be eligible to enter a catalog, get retrieved for the shopper’s request, make the shortlist, win the seller choice, and be purchasable. Different evidence covers each stage, and most confident advice about “how ChatGPT ranks products” quietly mixes them up.
Three kinds of evidence are worth having. Platforms document their inputs, but not the weights. Controlled studies test what moves a model’s pick inside a sandbox, but not inside a live assistant. Vendors observe real outputs and report associations. Anything else, such as a minimum star rating or a secret formula, is unknown.
Below is a ledger of what each kind of evidence supports, stage by stage. Every documentation source was read on October 27, 2026.
The four grades (a proposed scheme)
We sort every claim into one of four grades. The scheme is ours. It isn’t a standard; it exists to stop a lab result being quoted as a platform rule.
| Grade | What it means | What it can’t tell you |
|---|---|---|
| Documented | The platform’s own help or developer docs say it | Weights, thresholds, or how often an input decides the pick |
| Lab-tested | A controlled study changed one thing and counted the picks | What a deployed assistant does with your live listing |
| Vendor-observed | A company analysed real outputs or its own data | Cause, or anything nobody else has checked |
| Unknown | No adequate public evidence | Nothing. Treat confident claims here as folklore until someone tests them |
A documented input is real, but it isn’t a lever with a known size. A lab effect is real in its setup, but it can shrink, vanish or reverse on another model. Keep both in mind as you read the ledger.
The ledger, stage by stage
Stage 1: Can the product get in?
| Factor | Grade | What the evidence says | Limit |
|---|---|---|---|
| Shopify Catalog eligibility | Documented | Store on the Starter plan or higher and not in private mode, with an exception: Agentic plan stores that haven’t published an online store to a paid plan can still be included. Products need a title, at least one image, a price above 0 and an identifiable URL, and can’t be Unlisted or hidden from search engines | Gets you in. Says nothing about rank |
| Account standing | Documented | Shopify reviews standing over time: verified account, genuine sales to real customers, prompt and accurate fulfilment | No published thresholds |
| Which route each assistant uses | Documented | ChatGPT and Microsoft Copilot find Shopify products through Shopify Catalog. Google AI Mode and Gemini get them through the Google & YouTube channel, Meta through its own channel. Crawling is a separate route | Routes change; re-check the date |
| Blocking AI crawlers | Documented | A robots.txt block affects open-web discovery only. It doesn’t stop Catalog from sending your product data | — |
The plan and account rules come from Shopify’s Catalog requirements; the routes and crawler rule from Shopify Catalog and product discovery. Nothing on this stage is about copy. A store that fails it isn’t in the race, however good its descriptions are.
Stage 2: Is it returned for this request?
| Factor | Grade | What the evidence says | Limit |
|---|---|---|---|
| Query and buyer context | Documented | Shopify’s Global Catalog search takes a text query, an image or product IDs, plus buyer signals: country, region, postal code, language, currency and intent. Signed-in Shop buyers can get personalized results | Your test search without context isn’t your customer’s search |
| Shopify’s listing quality signals | Documented | Description word count, number of images, rating and review count from verified sources, variant and option completeness with stock, and store policies. Shopify adds that relevance also depends on popularity, customer engagement and brand recognition | Shopify says assistants often re-rank its results, so its preview is “a directional signal, not an exact prediction” |
| Google’s AI features | Documented | Supporting links must be indexed and eligible for a snippet. No new machine-readable files, AI text files or special markup are needed | Google Search only |
| Gains that don’t survive retrieval | Lab-tested | SAGEO Arena, a work-in-progress benchmark that puts retrieval back into the test, reports that earlier optimization methods can degrade retrieval and reranking | Not specific to shopping |
| Schema markup and AI citations | Vendor-observed | Ahrefs matched 1,885 pages that added JSON-LD against about 4,000 controls: +2.4% AI Mode and +2.2% ChatGPT citations, neither distinguishable from zero | Not randomized; page citations, not product picks |
The listing-quality list is the most concrete thing any platform has published about retrieval, and it comes with its own warning. Shopify’s search preview documentation says agentic storefronts “often re-rank results based on their own logic”. A good preview position is evidence about Shopify’s search, not about ChatGPT’s answer.
Stage 3: Does it make the shortlist?
This is the stage most advice is about, and the one with the most lab evidence.
| Factor | Grade | What the evidence says | Limit |
|---|---|---|---|
| Relevance to the query and the shopper’s context | Documented | OpenAI: a product appears when ChatGPT “perceives it to be relevant to your intent”, using the query and context such as Memory or Custom instructions, structured metadata from first and third parties, other third-party content, earlier model responses and safety policies. A stated budget makes price matter more | Inputs only, no weights |
| Google and Copilot inputs | Documented | Google Shopping’s “Top recommendations”: relevance, ratings, price and product features. Copilot: the prompt, likely engagement, available merchant data and other factors | One surface each |
| Stated facts instead of vague copy | Lab-tested | Every: one article went from 2 of 240 citations to 240 of 240 when its title and description said what it contained. ACES: one seller’s description edit gained between +0.32 and +14.89 points across six buyer models, significant in a third of experiments | Supplied candidates; size varies by model |
| Rewriting style | Lab-tested | E-GEO: simple rewriting improved mean rank by about half a position with GPT-5 and Claude rankers, and slightly worsened it with GPT-4o-mini and GPT-4.1. 11 of 15 “best practice” rewriting prompts did worse than the simple one | Products already retrieved |
| Ratings and review counts | Lab-tested | ABxLab: higher ratings raised selection by 30 to 80 points in 14 of 17 models. ACES: responses to price, rating and review count vary sharply by model | High ratings aren’t proof of fit or quality |
| Position in the list | Lab-tested | ACES: randomized position causes large, model-specific changes in what gets picked, even in plain JSON input | No universal “top slot” |
| Model and version | Lab-tested | ACES: one fitness tracker’s share rose from 45% to 77% between two Claude models (Sonnet 4 to Opus 4.5) and fell from about 25% to 6% between two GPT models (GPT-4.1 to GPT-5.1) | A model update can move picks with no change to your page |
| Labels and nudges | Lab-tested | ACES: “sponsored” tags generally penalized, platform endorsements rewarded. Bias Beware: social-proof wording often helped; scarcity and exclusivity wording often hurt | Not a reason to hide sponsorship or invent endorsements |
| Manipulative text | Lab-tested (security) | SafeGEO: the strongest realistic attack lifted a deliberately flawed product’s top-three placement from 12.7% to 95.9% on one model, against a truthful rewrite. Earlier work showed an optimized text sequence could push an ill-suited product up | A security finding, never a tactic |
| Overlap with Google Shopping | Vendor-observed | Peec: 83% of 43,000+ ChatGPT carousel products strongly matched Google’s top-40 organic Shopping results (March 2026) | A match isn’t proof of sourcing |
| Description truncation | Vendor-observed | Peec’s October trace analysis reports a 50-token description cut at one scoring stage | One vendor’s interpretation, not an OpenAI spec |
| Minimum rating, review count or a full formula | Unknown | No platform or study we read publishes one | — |
Read the lab rows together and one pattern holds: the agent is not an impartial spec calculator. Facts matter, and so do ratings, position, labels and which model happens to be running. A correct product record is necessary for an honest recommendation. It doesn’t guarantee one.
Our own simulations sit in this stage too. In our product-facts test, vague copy was excluded by every model, facts got the product considered, and it won where its facts answered a need the shopper had stated that the competing listings didn’t. Our article-title test found the same kind of effect for article titles. Both are supplied-snippet simulations, not live assistant results.
Stage 4: Which seller gets the sale?
| Factor | Grade | What the evidence says | Limit |
|---|---|---|---|
| Seller ranking in ChatGPT | Documented | Merchants are ranked on factors like availability, price, quality and whether they are the maker or primary seller | OpenAI says it expects this to change, including more personalization |
| Price shown | Documented | The first price shown typically comes from the first merchant listed and may not be the lowest. Price and shipping changes can take time to appear | — |
| Weights between these factors | Unknown | Not published | — |
A great product description doesn’t settle this stage. If three stores sell the same humidifier, availability, price and maker status decide who gets the sale, according to OpenAI’s help page.
Stage 5: Can it be bought?
| Factor | Grade | What the evidence says | Limit |
|---|---|---|---|
| Direct checkout settings | Documented | Shopify says direct checkout in AI channels is active where supported, and the Agentic section shows which channels offer it and whether you’re eligible. OpenAI says some eligible products and merchants get an Instant Checkout option | Eligibility varies by channel and store |
| Checkout flag in Catalog search | Observed | In our unpersonalized snapshot of 162 searches, the native checkout flag was false on all 1,620 first returned variants | One response per search; doesn’t show whether anyone could buy |
| Manipulation before payment | Lab-tested (security) | A 2026 study of agent payment flows reports credential, cart-mismatch and upsell attacks succeeding in 90%, 56% and 73.3% of its sample-agent runs | Specific attacks and test harnesses, not fraud rates |
What the checkout flag does and doesn’t mean is covered in our native checkout research.
What the evidence agrees on
Four things hold across the documents and the studies:
- Stated facts can change a choice after retrieval. Every’s article test, the ACES description edit, E-GEO’s rewriting and our own simulations point the same way. The size of the effect is not stable across models.
- Fit is relative to the shopper. OpenAI, Shopify and Microsoft all document shopper context, and ABxLab shows that preference instructions change how a model weighs attributes. A fact that wins for one brief can be irrelevant to the next.
- Agents respond to things that aren’t fit. Ratings, position, labels and model version all moved picks in controlled settings. So did manipulative text, which is why stated limits and honest labels matter.
- Stages don’t transfer. A citation result can’t establish a product’s rank, and Shopify Catalog order can’t establish ChatGPT’s final recommendation. Shopify says so about its own preview.
Folklore check
Claims you will hear, and which grade they actually have:
| Claim | Grade | What the evidence says |
|---|---|---|
| “Add schema and assistants will recommend you” | Vendor-observed, no effect detected | Ahrefs’ matched comparison found citation differences indistinguishable from zero. Keep accurate Product markup anyway; it doesn’t promise picks |
| “ChatGPT only reads the first 50 tokens” | Vendor-observed, one trace | One vendor’s interpretation of one stage. Front-loading key facts is worth doing; the number isn’t a rule |
| “You need four stars to be recommended” | Unknown | No published threshold. Labs show ratings matter, by different amounts per model |
| “You need an llms.txt file” | Documented against | Google says no special AI files are needed. Shopify also provides /agents.md, /llms.txt and /llms-full.txt URLs for stores, which you can customize |
| “Your Shopify preview rank is your ChatGPT rank” | Documented against | Shopify says agentic storefronts often re-rank its results |
| “Add ‘Updated 2026’ or ‘Bestseller’” | Lab-tested against | Every’s “(Updated 2026)” scored 0% against 2% for the original. In our Claude screening round, a “New 2026 / Bestseller” version of vague copy got 0 of 108 picks per category |
| “Persuasive copy always works” or “never works” | Lab-tested, mixed | Effects vary by model; some rankers penalize manipulation and some don’t |
One worked case, checked against the ledger
On October 26, 2026 we searched Shopify Catalog for “humidifier for dry winter air large room quiet top fill”. Position 2’s description said “21dB Quiet”. Position 4’s description said it “runs quietly enough to leave on overnight” and gave no decibel figure anywhere in its record. (Both listings, field by field.)
Here is what each grade lets us say about those two listings:
| Stage | What we can say | Grade |
|---|---|---|
| 1. Eligible | Both were in Shopify Catalog that evening, because both were returned | Observed |
| 2. Returned | Both were in the top 10 of one unpersonalized search. A shopper with a location or a Shop login might have seen a different list | Observed; Documented (context changes results) |
| 3. Shortlisted | A shopper who asks “how quiet?” can be answered from position 2’s record with a number, and from position 4’s only with an adjective. In our simulation, vague listings were almost never picked and listings that stated their fit facts were picked far more often. Whether ChatGPT showed either listing to anyone is unknown | Lab-tested; Unknown for any live assistant |
| 4. Seller | Unknown. We read the first variant only | Unknown |
| 5. Purchasable | Both first variants had the native checkout flag set to false in this response | Observed, one response |
The useful conclusion is narrow: position 4 made its listing harder to match on noise, and the published evidence says stating a fit fact is the part a merchant controls. Nothing here says position 2 outsold position 4, or that either appeared in a ChatGPT answer. Why a Catalog snapshot can’t answer that is set out in our note on Catalog research limits.
How to use the ledger
When someone tells you how agents pick products, ask three questions:
- Which stage is it about? Eligibility, retrieval, shortlist, seller or checkout. Advice about one stage rarely transfers to another.
- Which grade is it? A platform’s own statement, a controlled study, a vendor observation or nobody’s evidence at all.
- Where does it stop? Every row above has a limit. If the person quoting it left the limit off, put it back before you act.
Then spend your effort where the documented inputs and the lab evidence overlap and you have control: eligibility, accurate fit facts with units and conditions, honest ratings that travel with the record, current price and stock, and clear limits. Agent-ready product data shows how to build that record for one SKU.
Limitations
- The four grades and this ledger are a proposed scheme for sorting evidence, not an industry standard or a ranking model.
- Documentation states inputs, not weights. Living help pages change, and all of these were checked on October 27, 2026.
- Lab results come from sandboxes and supplied candidate sets with specific models. They don’t measure live assistants, and their effect sizes differ by model and version.
- Vendor reports were not reproduced by us.
- We can’t see what any assistant shows your customers. Our snapshot covers one route on one evening, and our simulations used invented products.
Your next step
Pick the one piece of advice you were about to act on this week. Find its stage and its grade in the ledger. If it’s documented or lab-tested and you control it, do it on one product and keep a dated copy of the record before and after. If it’s unknown, don’t spend a month on it.