We gave six AI models the same shopping task thousands of times: here are eight listings, pick the two best. One listing was a product we invented. Sometimes it described itself in vague copy, sometimes it stated plain facts. Everything else stayed the same.
The vague version was almost never picked by any model. The factual lip gloss was picked by every model, most of the time. The factual humidifier won on Claude, but on GPT-6.1-Sol and Gemini it won only when the shopper asked for exactly what it stated. When the ask was just “a quiet humidifier for my bedroom”, competing listings already stated the same or quieter numbers, and our product lost every time.
This is a supplied-snippet simulation. The products and their facts are made up, the models never searched or opened a page, and nobody bought anything. It tells you how six models chose among listings we handed them. It does not tell you what ChatGPT or Gemini show your customers.
The results in one table
Each cell is the number of held-out sessions in which the invented product was one of the two picks. Random picking would include it in 2 of 8, or 25%.
| Listing version | Claude (3 models) | GPT-6.1-Sol | Gemini 3.8 Flash | Gemini 3.1 Pro |
|---|---|---|---|---|
| Humidifier, vague | 3/180 (1.7%) | 0/60 (0%) | 6/60 (10.0%) | 3/54 (5.6%) |
| Humidifier, facts as a spec line | 134/180 (74.4%) | 20/60 (33.3%) | 19/59 (32.2%) | 15/55 (27.3%) |
| Humidifier, spec line + “not for” limit | 118/180 (65.6%) | 20/60 (33.3%) | 17/57 (29.8%) | 17/57 (29.8%) |
| Lip gloss, vague | 0/180 (0%) | 0/60 (0%) | 0/60 (0%) | 0/60 (0%) |
| Lip gloss, facts in the description | 168/180 (93.3%) | 60/60 (100%) | 49/60 (81.7%) | 53/60 (88.3%) |
| Lip gloss, spec line + “not for” limit | 168/180 (93.3%) | 59/60 (98.3%) | 32/60 (53.3%) | 45/60 (75.0%) |
Three things stand out.
- Vague copy was excluded everywhere. Six models, two categories, three kinds of shopper, and the vague versions never got anywhere near the 25% you’d get by picking at random.
- Facts were the entry ticket. Every factual version did far better than its vague twin on every model.
- How far facts carried the product depended on the model and on the shelf. The gloss facts nearly always won. The humidifier facts won on Claude, but on the other three models they won only under one kind of shopper brief.
The third point is the interesting one, and the reason this page exists.
What the listings actually said
Here is the invented product in its vague and factual versions. The brand name, Velmori, is made up, and so are the numbers. Nobody measured a Velmori humidifier, because there isn’t one.
| Version | Humidifier | Lip gloss |
|---|---|---|
| Vague original | “Whisper-quiet comfort, perfect for bedrooms. An elegant everyday essential that brings a calm, refreshing feel to your space.” | “Luminous beauty, effortless shine. A luxurious everyday essential with a silky feel and a polished finish, perfect for your signature look.” |
| Facts | “Noise: 28 dB on low · Coverage: up to 500 sq ft · Tank: 4 L · Runtime: up to 40 h on low” | “Clear, non-sticky lip gloss; no shimmer; fragrance-free; 5 mL tube.” |
| Facts + limit | The spec line, plus “Not for rooms over 500 sq ft.” | A spec line with the same facts, plus “Not for opaque colour coverage.” |
The price, star rating and review count stayed fixed across versions: USD 79.99 with 4.8 stars from 18 ratings for the humidifier, and USD 15.00 with 4.7 stars from 394 ratings for the gloss. Those were the medians of the seven real competitors on each shelf, so the invented product was priced and rated like a typical neighbour, with no edge it could buy.
The vague humidifier copy has the same problem as a real listing we found in Shopify’s agent catalog, which said it “runs quietly enough to leave on overnight” while the one beside it said “21dB Quiet”. An agent can compare 21 with 28. It can’t compare “whisper-quiet” with anything. (The two humidifier listings.)
Why the humidifier facts won on one shelf and lost on another
We ran each version under three shopper briefs:
- Detailed brief: a light sleeper with a 450 sq ft bedroom wants a quiet humidifier under USD 90, dislikes refilling daily, and wants evidence for room coverage, noise and runtime.
- One-line ask: “I need a quiet humidifier for my bedroom.”
- No shopper: the model is simply researching quiet humidifiers for bedrooms.
Split that way, the GPT and Gemini averages turn out to be two extremes averaged together:
| Humidifier spec line, by brief | Detailed brief | One-line ask | No shopper |
|---|---|---|---|
| Claude (3 models) | 59/60 | 32/60 | 43/60 |
| GPT-6.1-Sol | 20/20 | 0/20 | 0/20 |
| Gemini 3.8 Flash | 19/19 | 0/20 | 0/20 |
| Gemini 3.1 Pro | 15/15 | 0/20 | 0/20 |
Thirty-three percent looked like “barely better than chance”. In fact GPT and Gemini picked the product in every session with the detailed brief and in none of the others. The “not for” version showed the same split: every detailed-brief session on all three models (20/20, 17/17, 17/17) and none of the 120 one-line or no-shopper sessions.
The shelf explains a lot of this. Here is what the seven real competing listings stated, as we supplied them (a title, a description cut to 215 characters and three feature bullets). Store names are withheld.
| Listing | Price (USD) | Noise stated | Room size stated | Tank | Runtime |
|---|---|---|---|---|---|
| Competitor A | 119.99 | 21 dB | 700 sq ft | 11 L | 100 h |
| Competitor B | 39.99 | 28 dB | none | 4 L | 32 h in the description, 36 h in a bullet |
| Competitor C | 109.99 | 26 dB | none | none | none |
| Competitor D | 79.99 | 26 dB | none | 6.2 L | none |
| Competitor E | 69.99 | “quiet”, no number | none | 6.2 L | 62 h |
| Competitor F | 89.99 | none | none | none | none |
| Competitor G | 69.99 | none in what we supplied | none in what we supplied | none | none |
| Invented product | 79.99 | 28 dB on low | 500 sq ft | 4 L | 40 h on low |
Now read the picks against that table.
- With the detailed brief, the invented product was the only listing under USD 90 that stated a room size. The shopper had asked for evidence of coverage. Across the four model groups, the spec-line version was picked in 113 of 114 detailed-brief sessions.
- With the one-line ask or no shopper, the only requirement was “quiet”. On noise, our product was beaten: one competitor said 21 dB, two said 26 dB, and Competitor B said the same 28 dB at half the price with 310 ratings to our 18. GPT and Gemini included Competitor A, the 21 dB listing, in 113 of their 120 sessions. Claude included Competitor B in all 120 and put our product beside it in 75.
Competitor A’s description is the same “21dB Quiet” listing from our earlier catalog note.
The lip gloss shelf was different. Four of the seven competing snippets said “non-sticky”. A fifth said “without stickiness”, and several said “clear”, but none of them stated “no shimmer” or “fragrance-free”. One was vanilla-flavoured and two mentioned shimmer. The invented gloss was the only listing that answered every part of a clear, non-sticky, no-shimmer, no-fragrance brief, and every model found it.
Our interpretation (inferred, not tested): stated facts make a product eligible. They make it the pick only when they answer something the other listings don’t, or answer it better. Brand familiarity, review counts and price could also explain part of the pattern, and this design can’t separate them. We did not test changing the competitors or giving the invented product a quieter number.
The listing whose facts never made it into the snippet
Competitor G is a useful warning. Its full catalog description says it is “rated for single rooms up to 400 square feet and runs quietly enough to leave on overnight”. Both facts sat behind a long first sentence about the product name and nozzles, and our 215-character cut removed them. It was picked in 0 of the 1,062 held-out humidifier sessions across all six models.
That cut was ours, not a platform’s, so this is not evidence of how any assistant trims a listing. The lesson still holds: put the facts that decide the purchase in the first sentence, because you don’t control where someone else’s preview ends.
Two smaller findings
With a one-line ask, Gemini picked the vague humidifier more often than the factual one. Gemini 3.8 Flash picked the vague humidifier in 5 of 20 sessions and the factual one in 0 of 20 (Gemini 3.1 Pro: 3 of 20 against 0 of 20). The vague copy repeated the shopper’s own words, “quiet” and “bedroom”. The cells are small and the intervals overlap, but we report the result rather than hide it.
The “not for” limit held on Claude and GPT, and slipped on Gemini for gloss. For the humidifier, adding “Not for rooms over 500 sq ft” left the GPT and Gemini rates close to the spec-line rates. On Claude the pooled rate fell from 134 of 180 to 118 of 180. For gloss, the version with the limit matched the plain factual version on Claude (168 vs 168 of 180) and GPT (59 vs 60 of 60), but fell to 45 of 60 on Gemini 3.1 Pro and 32 of 60 on Gemini 3.8 Flash. The drop came from the one-line and no-shopper sessions (Flash with no shopper: 2 of 20); with the detailed brief, both gloss versions were picked 20 of 20 times on both Gemini models. For gloss, the limit version also changed the format, from a prose description to a spec line. So we can’t say the limit line caused the drop, and we can’t say it hurt.
None of our briefs needed opaque colour or a room over 500 sq ft. A limit exists to help an agent turn down the wrong buyer, and we never tested that. Keep truthful limits on your listings; this test is no reason to remove them.
How the test worked
We followed the method Every used to test article headlines: a fresh session for every answer, a fixed set of supplied results in a freshly shuffled order, and a count of how often the target made the pick. (Every’s experiment.)
- The shelf. For each category, seven real competing listings from a dated Shopify catalog snapshot, plus our invented product. Competitor text was held fixed, character for character, across every session.
- The task. Pick the two listings you’d recommend, returning their IDs. Listings got new IDs and a new random order in every session, so neither position nor ID gave away the target. No tools, no web, no page opening, low reasoning effort.
- Screening, then a fresh confirmation. In screening, three Claude models (Haiku, Sonnet and Opus 5.5) tested seven versions per category: 1,512 valid sessions. We chose two factual finalists per category before running the confirmation. The confirmation used 1,080 new Claude sessions (original plus the two finalists, three briefs, three models, 20 each). This page reports confirmation numbers only, so selecting finalists can’t inflate them.
- The replication. We then ran the same frozen shelves, prompts and finalists on GPT-6.1-Sol, Gemini 3.8 Flash and Gemini 3.1 Pro, with 360 planned held-out sessions each. Valid sessions: 360, 356 and 346.
- Failures. Failed or malformed replies were not repaired, rerun for a better answer, or counted as “not picked”. They were excluded, which is why some cells are under 60. Gemini 3.1 Pro hit 160 credit-limit rejections, retried once each as the protocol allowed, and 14 truncated replies. Gemini 3.8 Flash had 4 truncated replies. The first Claude launch sent 1,361 requests that the provider rejected for a schema setting before any model answered; we stopped it, fixed the schema and logged every rejected call.
- Position check. Shuffling doesn’t remove position effects; it spreads them out. Our first-half vs last-half diagnostic gap for held-out picks was 6.3 points on Claude, 0.5 on GPT, 13.4 on Gemini Flash and 2.5 on Gemini Pro. These are diagnostics, not corrections.
- Intervals. Confidence intervals are Wilson 95%. They describe repeated prompts to these models, not a sample of shoppers, stores or all AI agents.
- Spend. The three Claude experiments, including two tests not reported here, cost US$17.03 in known API charges (at most US$17.41 counting calls with unknown cost). The GPT and Gemini replication cost US$5.68 known (at most US$9.74). Total: about US$22.71 known.
The protocol, hypotheses and sample sizes were written down before any model was called.
What this does not show
- It doesn’t show live visibility. A supplied shelf guarantees the product a place. Real assistants first have to find the product, then rank, read and recommend it, and pick a seller. We tested only the last choice among eight.
- It doesn’t measure ChatGPT, Gemini or Google AI Mode. We called models through APIs. OpenAI says ChatGPT picks shopping results by relevance to the query and the shopper’s context, which our briefs imitate only roughly. (Shopping with ChatGPT Search.)
- It doesn’t forecast your lift. 74% is a share of simulated sessions for an invented product on one shelf. It is not a click rate, a conversion rate or a promise.
- It doesn’t show that rewording works. The factual versions disclosed facts the vague versions didn’t contain. If your page already states the facts, polishing the sentence is a different, untested question.
- It doesn’t generalize across models. We tested six models. Larger controlled studies also find that choices differ sharply between models and model versions. (What Is Your AI Agent Buying?; E-GEO.)
- It doesn’t test fit. No brief asked for something the product couldn’t do, and no model could answer “none of these”.
Test it on one of your own products
The useful takeaway isn’t “add facts and win”. It is narrower, and you can check it this afternoon:
- Write the shopper’s brief. Take the question customers already email you, such as “is it quiet enough for a bedroom?” or “will it cover 450 sq ft?”.
- Line up the shelf. Put your listing next to the four or five products a shopper would compare it with, using only what each listing states up front.
- Mark each need. For each part of the brief, note whether your listing states it with a value and a unit, states it vaguely, or doesn’t state it. Do the same for the competitors.
- Find the fact only you state. If competitors already state a better number for the main need, more adjectives won’t help. Look for a true, verifiable fact you can state that they don’t: coverage, runtime, what it’s not for.
- Write units the way a buyer would compare them. “28 dB on low” beats “whisper-quiet”. Units also have to match: in our own catalog research, our matcher missed “−30 °C” on two snowsuits because it was looking for “minus 30”. (What the catalog records show.)
- Change one fact at a time and re-check against the same shelf. That loop is laid out step by step in our agent commerce optimization guide.
Never invent a decibel figure, a runtime or a “fragrance-free” claim to fill a gap. Our test product could state anything because it doesn’t exist. Yours can only state what you can back up.