OpenAI Crawlers Can Update Its User Agent String

OpenAI made a number of explanation changes to its crawler thief documentation to opportunity that type number of the user-agent drawstring whitethorn beryllium updated and it whitethorn usage a user-agent drawstring pinch an further robots.txt marker.

The illustration fixed of the user-agent drawstring successful the doc, OpenAI added "the type number whitethorn change."

Plus, OpenAI added a conception to say:

When fetching robots.txt files, we whitethorn usage a user-agent drawstring pinch an further robots.txt marker. The marker helps tract owners much easy separate requests for the robots.txt record from requests for different resources, particularly erstwhile their logs do not see paths: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, for illustration Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

I don't deliberation immoderate of this is really caller but it was updated successful the thief documentation, truthful I wanted to make judge you each saw this.

Here are the changes documented from the erstwhile type I checked:

Forum chat astatine X.