Google hunt results and OpenAI’s Responses API together established 26.2% of the pages ChatGPT cited in a preprint’s tests of the GPT-5.5 Instant setting. They established 48.7% of the sites rearward those citations.
Figures in this paper were posted on Oct. 11 by researchers at the National University of Singapore and Xi’an Jiaotong University. It included four versions of ChatGPT, from GPT-5.2 Instant to GPT-5.5 Instant, alongside alongside the bulk of data gathering from March 29 through July 29, 2026. The two stand-ins were capable to be queried without having to open ChatGPT; the researchers compared the two stand-ins’ results for similarity to the pages cited by ChatGPT’s web app.
How The Researchers Watched ChatGPT
Researchers recorded the stream of data ChatGPT’s web use received during its penning process, by record what was happening in their own signed-in sessions. In this data stream you can see the searches ChatGPT performed, which pages resulted from the searches and the pages ChatGPT chose to be sources. A leaf was counted as cited if ChatGPT selected it as a source; i.e., the two pages that are individually listed as their own nexus and pages in groupings included as a sole choice as a source.
There were 80 questions. These 80 questions originated from Search Arena. This is a community repository of genuine person interactions alongside search-enabled chatbots/AI models. There were two languages used in these 80 questions: 40 English and 40 Chinese. Four types of requests were made: recommendation, fact lookup, data synthesis and analysis. Each of the 80 questions was run 20 times on GPT-5.3 Instant and GPT-5.5 Instant. GPT-5.2 Instant and GPT-5.4 Thinking had small questions run because of being decommissioned previous to completion of data collection.
To create the substitutes, researchers retrieved up to five pages of Google’s integrated results per inquiry through SerpAPI for all of the 80 questions. On average they got concerning 43 results per question. For all question, researchers additionally sent 20 queries to gpt-5.3-chat-latest and gpt-5.5 via OpenAI’s Responses API, alongside web hunt enabled. All of this was done inside three days of operating ChatGPT for the exact identical query. The document is an arXiv preprint. arXiv does not do equal assessment of papers.
Stand-Ins Matched Sites More Often Than Pages
Combined, Google’s results and the API established 28.3% of the pages GPT-5.3 Instant cited and 26.2% of the pages GPT-5.5 Instant cited. In conditions of sites, the identical blend established 50.3% and 48.7% of all cited sites (with subdomains being treated as distinct sites). The document calls a leaf cited if it has been cited on ChatGPT in at smallest one run of the 20. The percentages are afterward the averages throughout the 80 questions.
The Google hunt results solitary accounted for 9.4% and 6.4% of cited pages, and 26.1% and 22.8% of cited sites. The API’s complete origin catalog accounted for additional than that, showing 22.9% and 22.5% of all cited pages. The pages the API cited in its own answers covered less; however, whenever comparing them to ChatGPT’s citations, we can see that they were a nearer match. When looking at averages complete the 80 questions, GPT-5.5 Instant additionally cited 23.3% of the pages the API cited. This is higher than the percent of API-cited pages that were cited by GPT-5.3 Instant (at 16.4%).
ChatGPT Rewrote Questions And Changed Sources Between Runs
OpenAI’s assistance page explains that whenever ChatGPT hunt utilizes an external provider, it normally transforms a query into one or additional particular hunt terms. In the two GPT-5.3 Instant and GPT-5.5 Instant runs in which searches could be seen, ChatGPT never queried using the exact wording of the first question. However, in GPT-5.2 Instant, it did so in 5.5% of all runs.
Among the runs that included apparent searches for questions written in Chinese, GPT-5.3 Instant (67.0%) and GPT-5.5 Instant (69.6%) used lone the English language. Additionally, in February, I discussed results from Peec AI that established that in its tests, 43% of fan-out queries for non-English prompts ran in English.
Additionally, there was assortment in results between idiosyncratic runs. For example, 44.5% of the distinct pages returned for a stated inquiry were shown in lone among the 20 GPT-5.5 Instant runs. Similarly, 40.8% of the distinct pages returned for a stated inquiry were shown in lone among the 20 GPT-5.3 Instant runs. The flat of overlap between runs was greater at the flat of location than at the flat of page. Also, as noted by the authors, during pages that appeared in additional of the another runs connected alongside a particular inquiry had higher likelihoods of being referenced, they noted that this does not signify that citing is influenced by the frequence of appearance of a stated page.
I earlier covered the CiteChoice preprint concerning how rerunning identical inputs flipped 15% of cite-or-not decisions. Here, two gathering sessions transitioned to another example following operating for 14 and 16 runs respectively, and the authors hence propose checking which example generated all response.
Why This Matters
In the tests described here, neither of OpenAI’s API results nor of Google’s hunt results showed ChatGPT’s cited pages exactly as they were cited. This would average that an OpenAI API or Google ranking-based test would discover a larger portion of the identical site(s) than of the identical page(s).
The authors’ suggestions for how to detect are to detect at the two the flat of idiosyncratic webpages and of sites, run the identical immediate multiple times, and obtain into document the fact that ChatGPT rewrites your prompts into its own searches. Check if any difference of webpage-level results from a tracking tool versus what ChatGPT reports uses data collected through the API or immediately through the ChatGPT application. The document didn’t test any business tracking tool.
Looking Ahead
Every environment in the document predates GPT-5.6 and GPT-6. OpenAI’s example assistance page notes that GPT-6 is rolling out in Chat to eligible Plus, Pro, Business, and Enterprise users, during GPT-5.6 Luna is becoming the default for Free and Go users.
Around the GPT-5.5 alter in May, Sistrix data showed day-to-day assortment in German-language ChatGPT citations jump to 47%, against a normal 1% to 2%. The paper’s authors say monitoring strategies have to be re-evaluated whenever the example rearward ChatGPT changes. Until the difference is repeated on GPT-6, the paper’s figures depict the models that were current from March to July.
Featured Image: Accogliente Design/Shutterstock
OpenAI wordmark: OpenAI.
