Build the dashboard with one row per product, request and route, and keep three groups of fields apart:

  1. Visibility: what came back, as k of n runs under recorded conditions.
  2. Factual fidelity: which required facts were supported, wrong, words only or unknown, and in which field.
  3. Commercial: orders, sessions and conversion from your own analytics.

Every problem cell links to its evidence and opens an action with an owner, the proof needed and a next-check date. Don’t blend the three groups into a readiness score. A single number hides which fix to make and who should make it. This page gives a proposed layout, a data dictionary, the rules that turn a cell into a task, and two real catalog rows filled in.

Where the inputs come from

A dashboard doesn’t collect anything. It arranges what three other routines produce:

Input Produced by Lands in
Test cases: situations and hard constraints A shopping-intent panel Row identity
Repeated checks and their conditions The monitoring ledger and manual ChatGPT protocol Visibility
Requirement-by-requirement states and the next fix The one-product audit Factual fidelity, action queue
Orders, sessions, conversion, returns Your store analytics Commercial

If one of those routines isn’t running, its panel shows not checked. That isn’t a zero, and it isn’t green.

The layout

Four zones, top to bottom. The top three are read-only summaries of evidence. Work only gets created in the fourth.

Zone What it shows What it never shows
1. Conditions strip Market, buyer context, route, check dates and how many runs sit behind each cell Any result
2. SKU table One row per product, request and route. Visibility, fidelity and commercial columns grouped under separate headers A total, an average across routes, or a grade
3. Platform indicators Shopify’s own listing quality reading, in its own column and labelled as Shopify’s Blended into your columns
4. Action queue Each open action: the cell that triggered it, the change, owner, proof needed, status and next check Actions without an evidence link

Zone 3 deserves a note. Shopify’s listing quality indicator is a platform composite. Shopify says it breaks into description completeness, measured by word count, plus image coverage, product reviews, variant completeness and policy completeness (Managing agentic storefronts, checked October 27, 2026). It’s worth showing, because Shopify built it and you can act on it. But a description can pass a word count and still lack the one figure a shopper asked for. Keep it in its own column, so nobody mistakes it for your fidelity reading.

The data dictionary

Row identity and conditions

Field Definition Allowed values
sku_variant The exact variant checked Variant name with options
case_id The panel test case ID from the panel
route Where the result came from Shopify Catalog search; ChatGPT; another named assistant; crawled page
market Country set or used Country code, or “none set”
context Account and personalization state signed out / temporary unpersonalized / memory on / no buyer context
last_checked Date of the newest evidence behind the row Date
evidence_link Where the saved records are File or folder link. Required

Visibility

Field Definition Allowed values
runs Valid runs behind the cells, excluding failures Integer
returned_k Runs where the product came back in any form Integer, shown as k of runs
card_k Runs where it was shown as a product card Integer, k of runs
merchant_rank Your position in the merchant list, per run Numbers or “absent”, listed, not averaged
failed_runs Runs lost to no search, errors, limits or location problems Integer, with failure types
catalog_position Position in a Shopify Catalog search or preview, if checked Number, “not in top 10” or “not checked”

Factual fidelity

Field Definition Allowed values
requirements Hard constraints in the case Integer
supported / ruled_out The record states a value that meets, or fails, the requirement Integers
words_only / two_values / unknown No checkable value; conflicting values; nothing stated Integers
generated_only Requirements answered only in fields Shopify marks Inferred Integer, with the field named
wrong_fact_seen Any run in which an assistant stated a fact about you that’s false yes / no, with a link
condition_missing Published numbers without their setting, distance or method Integer

Counting rules for these fields, including what goes in the denominator, are in our coverage metrics guide.

Commercial

Field Definition Allowed values
channel_orders / channel_sales From Shopify’s agentic storefront analytics, per AI channel Number, date range, channel
channel_sessions / channel_conversion Online store sessions and conversion from that channel Number, date range
fit_returns Returns whose reason matches a case’s hard constraint, such as “too loud” Integer, from your returns data

Shopify says its agentic storefront reports cover sales, orders, online store sessions and conversion for each AI channel. Sales include both referral-based and direct-checkout sales, and the reports aren’t yet available for headless stores. Commercial fields sit at product or channel level, not case level, because a sale doesn’t carry the request that led to it. Never divide them by visibility counts to make a “conversion from visibility” figure. The two numbers come from different populations.

Action

Field Definition Allowed values
trigger The cell and value that opened the action e.g. “fidelity: words_only, noise”
change The specific edit or investigation One sentence
owner One named person Never a team
proof_needed The document the change depends on Manual, test report, spec sheet, channel setting
status Where it stands open / waiting on proof / shipped / verified / closed: no change needed
next_check When someone re-reads the evidence Date. Overdue cells are flagged

From cell to owned action

These are the triggers we’d start with. Each closes only when a new evidence row shows the change, not when someone says it’s done.

When this cell shows Open this action Usual owner Next check
wrong_fact_seen = yes Trace the source of the false fact and correct it Product-data owner Same day, then on the next scheduled run
two_values > 0 Settle the value at the source; correct every copy Product-data owner After the edit reaches the record
words_only on a hard constraint Replace the adjective with value, unit and condition, from a document Copy owner, with proof from the supplier Next scheduled check
generated_only > 0 Confirm or correct the generated value; publish it in your own text with its source Product-data owner Next scheduled check
unknown on a hard constraint Find the value or measure it. If it’s a limit, publish the limit Product owner When the proof arrives
returned_k falls on two dates and your control product doesn’t Check eligibility, channel settings and recent edits Store owner One week
Price or stock shown doesn’t match yours Check your feed and channel sync, then re-check Store operations 48 hours, then next scheduled check
ruled_out on a case you expected to lose Nothing. That’s a correct outcome — Next scheduled check

The price row needs patience. OpenAI says ChatGPT gets prices from third-party providers and that updates can take time to show (Shopping with ChatGPT Search). It doesn’t say how long. Record the time of each mismatch, and escalate only if it persists.

Why there’s no score column

We built a score once. It gave every search a gap score from 0 to 100, and when we audited it, about 65% of the points it awarded came from review depth, which was designed as only 25 of 100. Missing review fields had been read as zeros. We retired it and published the audit.

Three rules keep the same mistakes off a dashboard:

  1. Show k of n, never a percentage without its n. “Returned in 2 of 3 runs” can be checked. “67% visible” can’t.
  2. Unknown stays unknown. In our 1,620-result catalog snapshot, the product rating was absent on 440 and none carried an explicit zero (missing review data). A blank cell renders as “not stated” or “not checked”, never as zero or red.
  3. Never average across routes, markets or contexts. A Shopify Catalog position and a ChatGPT run are different measurements. Put them side by side, not in one number.

The test for any column: if it turns red, can you say who fixes what? If you can’t, it’s decoration.

A worked example: two humidifier rows

Imagine one merchant who owns two of the humidifiers that came back for “humidifier for dry winter air large room quiet top fill” in our Shopify Catalog search of October 26, 2026: the listings at positions 4 and 9. The merchant is a labelled hypothetical. In reality those listings belong to different stores. The record facts are real. No assistant checks or sales data exist for these rows, so those cells say so.

Conditions strip: route Shopify Catalog search; market none set; no buyer context; 1 run each, on October 26, 2026; re-read October 27.

Field Position 4 Position 9
catalog_position 4 of 10 9 of 10
runs / returned_k 1 / 1 of 1 1 / 1 of 1
ChatGPT fields not checked not checked
requirements 4: humidifier, large room (at least 450 sq ft), quiet (decibel figure), top fill Same 4
supported 2: humidifier, top fill 2: humidifier, top fill (title)
ruled_out 1: room, “up to 400 square feet” 0
words_only 1: noise, “runs quietly enough to leave on overnight” 1: noise, “low noise level”
unknown 0 1: room coverage
generated_only 0 (no generated metadata on the record) 1: noise, “26 dB” in the spec list
Commercial fields merchant’s own data, not available to us merchant’s own data, not available to us
Shopify listing quality not seen not seen

The same search makes the noise gap concrete. Four of the ten listings stated a decibel figure in their description, and two more, position 9 among them, had one only in Shopify-generated metadata.

Action queue:

Trigger Change Owner Proof needed Next check
Position 9, words_only + generated_only: noise Replace “low noise level” with the measured figure, setting and distance Copy owner Manual or test report. Shopify’s generated “26 dB” isn’t proof Re-read the catalog record after the edit; date it
Position 4, words_only: noise Publish a measured decibel figure with its setting and distance Copy owner Manual or test report Same
Position 9, unknown: room coverage Publish rated coverage from the spec, or state it’s for small rooms Product owner Manufacturer spec sheet When the proof arrives
Position 4, ruled_out: room None. “Up to 400 square feet” is a true limit for a 450 sq ft brief — — Next scheduled check

Two things to notice. Both products need the same kind of fix, a decibel figure with its condition, but for different reasons: one has only an adjective, the other has a number Shopify generated and the merchant never wrote. And position 4’s “ruled out” generates no work. A dashboard that turned that cell red would push someone to soften a true limit.

These are the rows of a dashboard on day one: one route, one run, nothing yet from ChatGPT. The ChatGPT columns fill in once the manual protocol has run, and the commercial columns come from the merchant’s own Shopify reports.

Keep it honest over time

  • Date every cell. A cell older than its next-check date shows as overdue, not as current.
  • Show the run count beside every visibility figure. One run and fifteen runs shouldn’t look the same.
  • Keep closed actions visible for one cycle. The next check either confirms the change or reopens the action.
  • Log every product-data edit with its date, so movement can be read against what changed.

How we checked

The worked rows come from a Shopify Global Catalog search we ran on October 26, 2026 with Shopify’s catalog command-line tool, top 10 results, no buyer context. On October 27 we re-read the saved records for positions 4 and 9 and checked every quote and state above against the field it came from: title, description with HTML removed, and Shopify-generated metadata. The 450 sq ft room threshold is the example brief from our coverage guide.

We read Shopify’s agentic storefronts management page, Shopify’s Global Catalog documentation and OpenAI’s shopping help page on October 27, 2026. We haven’t seen Shopify’s admin reports or indicators for any store. Checked on 2026-10-27; re-check before acting.

Your next step

Make a sheet with the four zones above and one row for your best-selling product and its most common customer request. Fill the fidelity fields from a one-product audit, mark every visibility and commercial field you haven’t checked as “not checked”, and open one action with an owner, the proof needed and a next-check date. A dashboard starts with that one owned row.