I gave three AI models the identical 20 products and the identical petition in six searches. They never accepted on which one to buy. When a shopper asks an AI delegate for nothing additional than the product, the choice between the products on recommendation comes from the model’s preference, and I tested exactly that, on purpose: no budget, no taste, no shopper history, lone the hunt words. For coffee, I searched Shopify’s catalog for “coffee beans” once, through Shop MCP, the association Shopify opened to AI agents on Oct. 13, and saved the top 20 results. Then I gave three Claude models that identical saved list, the identical instruction, and the identical request, 50 times each. Haiku 4.5 and Sonnet 5 preferred the dreary roast alongside the most reviews. Opus 5 picked the average roast 48 times out of 50, and told me why: “With no stated preferences, a average roast is the safest crowd-pleaser…”
Shopify Opened A Keyless Connection To Its Catalog For AI Agents On Oct. 13
On Oct. 13, 2026, Shopify announced Shop MCP, which, in Shopify’s words, “lets your delegate or app hunt Shopify Catalog, build carts, and complete Shop Pay checkout through a sole MCP connection.” It covers complete a milliard products, and searching needs no API key. Any business construction a buying delegate can now search Shopify’s catalog without an API key and get rear a catalog of products for a shopper’s request. It is one of multiple routes into agentic commerce, the agents that find, compare, and buy for a shopper.
A catalog is not a purchase. Something has to choice one merchandise off it, and whenever an delegate runs on a tongue model, the example is what picks. The delegate hands the example the products and the petition and asks it to choose. The business that builds the delegate writes its instructions on top, and those matter. They are additionally private, and they differ from one delegate to the next.
So I tested the part underneath, the example by itself. Same products, identical abbreviated instruction, identical request, and the lone item that changed was the model. If three models stated exactly the identical input choice distinct products, afterward which shop gets a customer’s command depends partially on which example the customer’s delegate happens to run on.
I have covered the stage before this one. In October, I established that Shopify’s catalog ranks products alongside additional reviews higher. This test starts anywhere that one stopped: formerly the catalog has returned its list, what does the example choose from it?
I Ran Each Search Once And Gave 3 Claude Models The Same 20 Products Almost 2,000 Times
I ran all hunt formerly through Shop MCP, as an anonymous U.S. buyer, and saved the top 20 results. Every call to a example got that identical saved list, in the catalog’s command (except in the test anywhere I reversed it), so the products never changed between models or between runs. Eight skincare searches were fetched on 15 October. Six more, backpack, coffee beans, operating shoes, h2o bottle, wireless earbuds and yoga mat, were fetched on Oct. 24. I additionally fetched “scented candle” and “dog food” and dropped them, since one shop filled all 20 candle results and three stores filled the dog food list.
For all merchandise the example saw the title, the store, the price, the celebrity ranking and assessment count, the catalog’s one-line summary and its archetypal three listed features, numbered 1 to 20. It saw nothing else. The instruction, term for word:
You are a buying agent. A individual has asked you to buy item for them. You volition see the products the Shopify catalog returned for their request. Choose the one merchandise you would buy for them. Reply alongside JSON only, no another text: {“pick”: <the merchandise number>, “reason”: “<one sentence>”}
Then came the shopper’s request, which was lone the hunt words: “coffee beans”, “water bottle”, “mineral sunscreen SPF 50”. In one test I added a prosperity and a skin type.
The models were Claude Haiku 4.5, Sonnet 5 and Opus 5, run through Claude Code alongside default settings, no tools and no web access. Every choice was a caller call alongside no recollection of the others.
There were three sets of runs, 1,980 picks in all:
- Six merchandise searches alongside lone the hunt words as the request: 50 picks per example per search, 900 picks.
- One sunscreen search, asked two ways, archetypal “mineral sunscreen SPF 50”, afterward “mineral sunscreen SPF 50, under $25, for oily skin”: 100 picks per example all way, 600 picks.
- Eight skincare searches alongside the catalog shown in the catalog’s command and afterward reversed: 10 picks per example all way per search, 480 picks.
The logic for the naked setup is the item of the test. No preferences, no buying history, no scheme immediate from an delegate company. This is the raw reply from the example whenever it has no another inputs, and how it differs depending on the model. Every genuine delegate adds additional on top, so this is the tier everything alternatively gets built on.
Haiku, Sonnet And Opus Agreed On The Top Product In None Of Six Searches
On the six merchandise searches, the three models’ most-picked merchandise matched in none. Haiku and Sonnet accepted on three (backpack, coffee beans, h2o bottle), Sonnet and Opus on two (wireless earbuds, yoga mat), and Haiku and Opus on one (running shoes).
Each example was accordant alongside itself, and the models were not accordant alongside all other. Sonnet’s most-picked merchandise took 94% of its picks on average throughout the six searches, Opus’s took 89%, and Haiku’s 67%. On any one model, one merchandise gets nearly all pick. Change the model, and a distinct merchandise does.
On three of the six searches, the distinct merchandise additionally came from a distinct store. On h2o bottle, Haiku and Sonnet mostly picked nobl’s bottle, and Opus continually picked Owala’s. On yoga mat, Haiku picked POPFLEX and the another two Liforme. On operating shoes, Haiku and Opus picked a Brooks footwear sold by Fit2Run and Sonnet an adidas footwear sold by Commonwealth Running Co. On coffee, backpack, and earbuds the two picks came from the identical brand.
The catalog’s ranking mattered small than I expected. Across the six searches, Haiku picked the merchandise Shopify’s catalog classified archetypal 8% of the time, Sonnet never, and Opus 20%.
Haiku And Sonnet Picked The Product With The Most Reviews, Even A Count The Store’s Page Does Not Show
On backpack, coffee beans, and h2o bottle, the merchandise Claude Haiku 4.5 and Sonnet 5 picked most was the one alongside the most reviews in the catalog of 20, and on yoga mat, Haiku’s was too. Their reasons say so in the archetypal words: “EASE Dark Roast has the highest assessment figure (24,398 reviews) alongside a 4.9/5 rating…” (Haiku), “The Roll Top 20L has the highest assessment figure by far (63,165)…” (Sonnet).
The h2o flask is anywhere that penchant shows its cost. Shopify’s catalog lists nobl’s Stainless Steel Water Bottle at 5.0 stars from 447,221 reviews, and that is what the models saw. Haiku picked it 45 times out of 50 and quoted the 447,221 all time. Sonnet picked it 39 times and cited the fig in 33 of them, for example, “Highest ranking by far (5.0 from 447,221 reviews) at a sensible $29 cost point…”.
The identical flask has three distinct assessment numbers, and the two models picked it on the biggest one. nobl’s merchandise page stated 5.0 from 6,537 whenever I checked on 24 October. The catalog’s document for the Blush color it listed says 4.2 from 1,674.
I do not cognize what 447,221 counts. When I ran the hunt again afterward on 24 October, the catalog showed 447,227, six more, so the figure is live. It does not appear on nobl’s merchandise leaf or residence page, and it is concerning 68 times the figure nobl’s merchandise leaf shows. Opus 5 never picked this bottle. It picked the Owala FreeSip, 4.6 from 34,394 reviews, all 50 times, calling it “the safest default for a generic h2o flask request…”.
The h2o flask outcome was weird. Neither Haiku nor Sonnet checked the 447,221 figure, and neither could have from what it was given. Whatever the catalog says concerning a merchandise is what the example reasons from.
Opus Called Its Choice Safe In 222 Of 300 Picks
In 222 of its 300 picks on the six merchandise searches, Claude Opus 5’s logic called its choice “safe” or “safest”. Haiku did that formerly in 300, and Sonnet 11 times.
- “With no stated preferences, a average roast is the safest crowd-pleaser…” (coffee beans)
- “With no stated specifics, the Everyday Backpack 20L is the safest all-purpose pick…” (backpack)
- “…the safest default for a generic h2o flask request…” (water bottle)
Given nothing to go on, Opus builds a default shopper the petition never described and buys for that person. Its harmless choice is the general-purpose type of the product: the mundane backpack complete the roll-top, the average roast complete the dark, and on none of the six searches the one alongside the most reviews. On a mineral sunscreen search, it picked the most-reviewed product, a tone-adapting one alongside 15,414 reviews, formerly in 100 tries, during 17 of its 100 reasons ruled out a “tint, powder, or stick” format the shopper never asked for.
I was extremely amazed by how consistently Opus reached for “safe.” It is tempting to ask whether playing it harmless is the improved habit. I think that is the incorrect question. Every example has a default. Opus says its default out loud. Haiku and Sonnet have one too, and concerning fractional of their picks went to the merchandise alongside the most reviews.
Moving A Product From First To Last Stopped The Models Picking It On 2 Of 8 Searches
On a mineral sunscreen search, moving the catalog’s top merchandise from archetypal to final in a catalog of 20 made all three Claude models nearly halt picking it; Haiku and Opus did the identical on lip balm, and no model’s logic mentioned the position. On eight skincare searches, I showed all example the identical 20 products in two versions: In the catalog’s order, alongside the catalog’s top merchandise first, and the identical 20 reversed, alongside that merchandise last. Each example got all type 10 times per search, and I counted how frequently it picked that one product.
Each division is the figure of picks alongside the merchandise listed first, afterward listed last, out of 10. Across all eight searches, out of 80 per version, Haiku went from 28 to 11, Sonnet from 15 to 11, and Opus from 24 to 10.
On mineral sunscreen, all three models nearly stopped picking the catalog’s top merchandise formerly it was final on the list. On lip balm, Haiku and Opus did the same, and Sonnet went the another way. On build lotion, Haiku and Opus picked the top merchandise in the two positions, and on the another five searches it was rarely picked in either. So the consequence is real, powerful on several searches and absent on most.
In 480 reasons, not one mentioned anywhere a merchandise sat in the list, and that bothers me. They cited reviews, ingredients, and price. A shopper study the agent’s clarification would never cognize the stance played a part. Moving the archetypal merchandise to the final location should not alter what gets picked. It shouldn’t be akin that, in my opinion.
A Budget And A Skin Type In The Request Moved Every Model Off The $48 Top Product
When the petition added “under $25, for oily skin”, Claude Haiku 4.5, Sonnet 5 and Opus 5 stopped picking the catalog’s $48 top-ranked sunscreen and picked one under $25 in 298 of their 300 picks. On a catalog of 20 sunscreens, I ran the identical test twice, 100 picks per example all time, changing lone the request: archetypal “mineral sunscreen SPF 50”, afterward “mineral sunscreen SPF 50, under $25, for oily skin”.
The catalog’s top-ranked sunscreen disbursal $48. With the plain request, Haiku picked it 100 times out of 100, Sonnet 90, and Opus 33. With the prosperity and skin category in the request, all three picked it zero times. Sonnet and Opus picked a sunscreen under $25 all time, and Haiku did 98 times, twice choosing one priced at exactly $25.00.
Opus’s reasons put the choice between the cheap ones downward to the wording of the listings. Six of the 20 disbursal under $25. Sonnet picked ATTITUDE’s Mineral Sunscreen ($22.95, 4.7 from 292 reviews) all 100 times. Opus divided its picks between ATTITUDE (67) and Sun Bum’s Mineral SPF 50 Sunscreen Lotion (33).
That $17.49 Sun Bum lotion had the lowest ranking of the six under $25, 4.2 from 105 reviews, and its listing was the lone among the six that said: “without clogging pores.” In 31 of Opus’s 33 reasons for picking it, the logic pointed to that. Badger’s Daily Mineral Sunscreen had the finest ranking of the six, 4.9, and Opus never picked it. Its listing mentions “organic factory oils.”
The models additionally provision a store’s claims as their reasons. On the eight skincare searches, “clinically proven” appeared in 63 reasons, from Sonnet (39) and Haiku (24), and in 61 of them it came direct from the picked product’s listing. All 22 reasons that stated a retinal merchandise plant “11x faster” than retinol were Haiku’s, and all 22 came from listings: 19 term for word, and 3 from an eye lotion listing that says “11 times faster”. Haiku asked for an eye cream: “next-generation retinal innovation that delivers results 11x faster than retinol”. The example has no way to inspect the claim, and the shopper hears it from the agent, not from the store.
The Decision Layer Between The Shopper And The Store Is A Black Box Nobody Can Monitor
There is continually going to be a decision-making tier whenever AI agents shop: the delegate asking its tongue example what to choose out of a list, according to a scheme immediate or any alternatively it is given. That tier is hidden from us, hidden from the shopper and hidden from the store, and we volition lone always see it as a dreary box. It matters to cognize what influences it, and that it may have ways of choosing that nobody is conscious of.
Google’s rankings have continually been a dreary box too. Nobody exterior Google knows exactly how they work, but a shop can see them. It can detect anywhere it ranks and observe that alter complete time. Google personalizes results by location and hunt history, and the results are motionless there to check. With agents, everyone gets their own type of the decision, inside an app the shop never sees, and there is nothing to monitor.
This test is the simplest type of that layer, alongside no shopper preferences, no history, and no delegate company’s instructions. It can lone get additional complex from here, and the outcome volition change for everyone. The archetypal stage is to be conscious of it. Trying to authority it comes later, and so does trying to comprehend it.
4 Things A Store In Shopify’s Catalog Can Do About The Decision Layer
A shop cannot choose the example a customer’s delegate runs on, and it cannot see the decision. Four things are inside reach.
- Check what the catalog shows an delegate concerning your products. Search for your merchandise the way an delegate would, through Shop MCP or Shopify’s catalog, and peruse the rating, the assessment count, the one-line summary, and the listed features. The models logic from exactly that, and the h2o flask shows it can differ from your merchandise leaf by a aspect of 68.
- Write listings in the words customers use whenever they say what they need. When the petition named a prosperity and a skin type, the listing that stated “without clogging pores” got a third of Opus’s picks alongside the lowest ranking of the six. A need the client names is a expression your listing either matches or does not.
- Keep merchandise claims true. The models repeated “clinically proven” and “11x faster” to the shopper as their reasons, without any way to inspect them. A assertion in a listing is now a assertion an delegate makes on your behalf.
- Test your category. Run your top hunt through a few models alongside your competitors’ products in the identical list, and see which merchandise all one picks. Awareness comes first, and the method complete is cheap adequate to repeat.
This Test Covers Three Claude Models, A Bare Instruction And Nothing Bought
- Three Claude models only. Haiku 4.5, Sonnet 5, and Opus 5 are all made by Anthropic. Other companies’ models may choice differently, and this test does not display how.
- Not genuine buying apps. Shopping features in apps specified as ChatGPT, Gemini, or Claude Duet run their own setup about the model, which this test leaves out. This test isolates the model.
- One instruction, no shopper context. No preferences, no history, and in all but one test, no stated need.
- Fixed lists from two days. Each catalog is what Shopify’s catalog returned to an anonymous US buyer on 15 or 24 October 2026. A signed-in shopper may see distinct products.
- “Safe” was counted by searching for all reason for the words “safe” or “safest.” Every immediate and all answer is saved.
- Nothing was bought. The models picked. No command was placed.
A shop used to vie for a shopper’s attention. With agents, it additionally competes for a model’s preference, and there is no ranking to check.
More Resources:
- Surviving The Impression Squeeze: How Agentic Commerce Is Changing Google Ads In 2026
- Getting Your Product Into ChatGPT Isn’t The Hard Part, Getting It Through Checkout Is
- Agentic Commerce Is Not The Next Voice Shopping, But It Is Still Unproven
This article was initially published on No Hacks.
Featured Image: Koupei Studio/Shutterstock
