ChatGPT’s merchandise recommendations changed 80.2% erstwhile hunt was enabled, according to a study of 20,000 responses by Jeff Oxford, laminitis and CEO of Visibility Labs.
Oxford tested 1,000 product-recommendation prompts 10 times each pinch ChatGPT hunt enabled and 10 times pinch hunt disabled.
Only 19.8% of products recommended without hunt besides appeared successful recommendations generated pinch hunt enabled.
Search changed apical picks. Even products ChatGPT recommended astir often without hunt seldom carried over. Among products that appeared successful 100% of search-disabled responses, only 15.8% besides appeared erstwhile hunt was enabled.
- Oxford expected the astir consistently recommended products to stay communal erstwhile hunt was turned on. Instead, that group had the lowest overlap.
Source mentions tracked visibility. The study besides examined whether products mentioned successful ChatGPT’s cited sources appeared much often successful its recommendations. Oxford reported a 0.4 Pearson relationship betwixt cited-source mentions and proposal frequency.
- The study measured proposal wave pinch a “Visibility Score,” defined arsenic the percent of runs successful which a merchandise appeared for a fixed prompt. Products mentioned much often successful cited sources tended to person higher Visibility Scores.
- The study didn’t found that cited-source mentions caused products to beryllium recommended.
Search narrowed recommendations. ChatGPT responses pinch hunt enabled contained an mean of 5.2 products, compared pinch 6.2 erstwhile hunt was disabled.
- Across 10 runs of each prompt, ChatGPT returned an mean of 19 unsocial products per punctual pinch hunt enabled and 21.8 pinch hunt disabled.
Why we care. Search changed which products ChatGPT recommended, including products it named each clip erstwhile web entree was disabled. The findings propose products appearing successful cited sources whitethorn person greater visibility erstwhile hunt is enabled, though the study does not find whether cited-source visibility matters much than broader web visibility.
About the data. Oxford analyzed 1,000 product-recommendation prompts, moving each 10 times pinch hunt enabled and 10 times pinch hunt disabled. Product names were standardized truthful naming variations counted arsenic the aforesaid product. Because the study was observational, it did not found a causal narration betwixt cited-source mentions and proposal frequency.
The report. ChatGPT’s Product Recommendations Change 80.2% When Search is Enabled vs Disabled (Study of 20,000 Responses)
Dig deeper. AI proposal lists repetition little than 1% of the time: Study
English (US) · 
Indonesian (ID) ·