Say you swapped “runs quietly” for “26 dB on the lowest setting” on your best-selling humidifier. Now you want to know if it worked.
Here’s the test: pick one product to change and a similar one to leave alone. Ask the same list of customer questions a few times before the edit and a few times after, the same way each time.
Then compare the two products. If the changed one moved and the untouched one didn’t, you’ve learned something real.
It takes about 20 minutes to set up, a few minutes a day to run, and two weeks from start to finish.
Why one check after the edit fools you
AI assistants don’t give the same answer twice. OpenAI says ChatGPT chooses shopping results from the question and the person’s context, such as Memory and custom instructions. Ask on Monday, ask on Tuesday, and you can get different products with nothing changed on your side.
So a single “before” and a single “after” mostly measures luck. You need repeats, and you need something to compare against, so you can tell your change apart from the assistant changing under you.
Here’s how small the real signal can be. In our simulation, where AI models picked from listings we supplied, a humidifier with vague copy was picked in 3 of 180 Claude sessions. Check it three times and you’d almost certainly see zero, and conclude it never shows up.
With its facts written as a spec line, the same humidifier was picked in 134 of 180. Live assistants also shift from day to day, which is why the repeats matter.
Set up the test on one page
Write these down before you touch anything:
| What | Example |
|---|---|
| The change | Description: “low noise level” becomes “26 dB on the lowest setting” |
| The test product | The humidifier you’re editing |
| The comparison product | A similar product you won’t touch for two weeks |
| The questions | 6 to 10 real customer questions (see below) |
| Where you’ll check | Shopify’s search preview, plus ChatGPT in a temporary chat, from the same country |
| How often | 3 runs per question, on 3 days before and 3 days after |
The comparison product is the part most people skip. Pick one in the same category that gets asked about in the same way, and don’t edit its price, photos or copy until the test ends. If it moves as much as your test product, the assistant changed, not your listing.
Write questions the fact should win, and ones it shouldn’t
Pull your questions from support emails, reviews and return reasons. Our question list guide shows how. Then sort them into three groups:
- Questions the new fact answers. “Quiet humidifier for a nursery.” This is where you expect to gain.
- Questions it shouldn’t affect. “Top-fill humidifier under $50.” If these move too, something else is going on.
- Questions you should lose. “Humidifier for a 600 sq ft living room,” for a bedroom unit. Being shown here isn’t a win. It’s a future return.
That last group earns its place. In our simulation, on GPT and Gemini, the factual humidifier was picked every time the shopper asked for the exact facts it stated, and never when the ask was just “a quiet humidifier for a bedroom”.
A fact wins the questions it answers, and your list needs to tell those apart from the rest.
Record the “before”, then make one change
Run every question three times on each of three days. For each run, write down:
- the date, and where you checked;
- whether your test product was shown, and in what position;
- whether your comparison product was shown;
- whether one of your other products took the spot instead;
- what the answer said about the fact you’re changing.
Save the answer text or a screenshot. “Shown in 2nd place, described as quiet” is a useful row. “Yes” isn’t.
Then make the change, and only that change. Keep a short log: the date, the product, the field you edited, the old text, the new text and the document that proves it (a manual or test report).
Leave price, images and reviews alone this week. If something else happens anyway, like a sale, a stockout or a new competitor, write it in the log.
Wait for the change to land, then check the same way
Before you re-run anything, make sure the new text has actually reached Shopify’s catalog. Search for the product in Shopify’s search preview under Sales channels > Agentic. Shopify says it shows the raw catalog results, though the AI apps re-rank them their own way.
Assistants then pick it up on their own schedule. FoundRate tells its customers to give a fix about a week, since AI shopping tools re-check stores every few days to a few weeks. So start your “after” runs about a week after the edit, then repeat exactly what you did before: same questions, same places, same country, three runs a day for three days.
Read the results
Count, per product and per question group, how many runs showed it. Then read it like this:
| What you see | What it means |
|---|---|
| Test product up on the “fact” questions, comparison flat | Your change probably helped. Keep it, and roll the same fix out to similar products |
| Both products moved the same way | The assistant changed, not your listing. Run it again later |
| Test product up on the “should lose” questions too | You’re being shown to the wrong shoppers. Make the limit clearer (“for rooms up to 300 sq ft”) |
| Test product up, but another of your products dropped out | Your products are trading places. See sibling products competing |
| Nothing moved anywhere | Try again in a week. If it’s still flat, the fact may not be what decides these questions |
“Nothing moved” is a normal result. It still tells you something: either the fact wasn’t the missing piece, or the assistants haven’t picked it up yet.
Also watch sales, separately
Shopify reports sales, orders, sessions and conversion for each AI channel, under Sales channels > Agentic. Pull the same date range before and after.
For most small stores those numbers are too small to move visibly in two weeks, so don’t let them decide the test. Keep them in the log anyway. If the product starts getting picked more, this is where it eventually shows up.
How we checked
We built this test from what we learned running our own simulation of AI picks, where the same listing with and without its facts gave very different pick counts. On November 4, 2026 we re-read OpenAI’s shopping help page, Shopify’s agentic storefronts help page and FoundRate’s FAQ. The humidifier example is the listing from our one-product audit.
Your next step
Pick the product you’re about to edit anyway, and one similar product you’ll leave alone. Write six real customer questions, including one it should lose, and run the “before” checks this week, before you change a word.