Robots.txt, as the net norm RFC 9309 defines it, governs which URLs a crawler may request, not what happens to the satisfied afterward, Google’s Gary Illyes stated in a Search Central video published Wednesday. To keep one AI crawler out during allowing others, he said, you have to mark that crawler’s user-agent token in the file.
The video is the second event of The Non-Canonical Cabinet, anywhere Illyes and Cherry Prommawin of Google Search Relations obtain edge-case questions from the community. Illyes is additionally co-author, alongside Mozilla’s Martin Thomson, of an IETF Internet-Draft that would provision robots.txt a norm way to province use preferences. Prommawin recommended penning non-English regulation paths in percent-encoded form for crawlers another than Google’s.
What Illyes Said About AI Training
The episode’s inquiry was whether robots.txt can authorize AI crawling of a website and keep that identical data out of generative example training. According to Illyes, “So as defined by RFC 9309, robots.txt is an admission penchant protocol.” He went on to say robots.txt “doesn’t dictate what a scheme does alongside the data following it has already crawled it.” If you desire to entirely obstacle a particular AI crawler from accessing your complete website, Illyes stated, you need to “target their particular person delegate tokens explicitly in the file.”
The event illustrated a fictional directive (e.g., No-AI-Chicken-Training) which was shown on-screen and would be ignored by a parser. Google’s robots.txt specification page states that the four elements supported are User-Agent, Allow, Disallow, and Sitemap; invalid lines volition be ignored by Google.
As for RFC 9309 specifically (Section 2.2.4), it allows crawlers to decide how they’ll procedure a document exterior the range of this protocol. Therefore, whether or not an additional row accomplishes item depends on the crawler interpreting it.
What Google-Extended Controls
Google-Extended is a robots.txt token alongside no person delegate of its own. Google crawls under its existing person agents, and Google’s list of average crawlers says the token is “used in a authority capacity.” It manages whether satisfied Google crawls can be used to train forthcoming Gemini models and for grounding in Gemini Apps and Grounding alongside Google Search on Vertex AI. The leaf says it doesn’t power a site’s inclusion in Google Search and isn’t a ranking signal.
Google-Extended’s definition as a use authority comes from Google’s records for its own products, fairly than RFC 9309. The hosts didn’t conversation it on air, but the video’s clarification links to the identical list.
The Draft That Would Add A Usage Rule
The current type of the IETF outline Illyes co-wrote, dated Aug. 19, would update RFC 9309, if approved, alongside a Content-Usage regulation for robots.txt and a matching HTTP header. Neither the event nor its clarification mentions the draft.
Under the draft, Allow and Disallow would continue to govern which URLs can be crawled. Content-Usage would afterward define preferences as to how crawled satisfied have to be used. Those preferences would lone connect to paths anywhere a crawler has approval to fetch content. A Content-Usage site of “train-ai=n” is added to the draft’s example robots.txt file, which allows crawling.
The outline is motionless a Working Group Document. At present, its IETF Datatracker admission shows that the Working Group has not yet submitted the document for assessment by the IETF’s Steering Group (IESG). The Working Group’s planned date for sending it to the IESG was Aug. 2026. The ‘train-ai’ Label referenced complete appears in a distinct Vocabulary Draft, dated Sept. 14. That document states that it reflects no agreement inside the Working Group.
Percent-Encode Non-English Paths
Cherry Prommawin’s direction for paths alongside non-English characters or emoji was that “you should use the encoded type in your robots.txt rules.” Google’s parser handles raw characters by encoding them, she said, but “other or older crawlers on the web power get confused.”
The on-screen example tagged a raw emoji way “Works for Google, power interrupt others” and its percent-encoded form “The Safe, Standard Way.” Google’s spec leaf says its parser treats raw UTF-8 and percent-encoded paths the identical way, and RFC 9309’s grammar allows UTF-8 characters in paths.
Why This Matters
The method you would need in robots.txt to bounds AI use of your satisfied during keeping your location apparent in Google Search depends on what bot or merchandise you are operating with. Disallow statements may forestall particular AI crawlers from accessing particular webpages. For Gemini, the documented authority of Google is Google-Extended and includes the two training and grounding. There currently is no norm way of expressing a training penchant autonomous of the bots allowed to crawl the webpage; one exists lone as a draft.
Websites that use URLs alongside languages another than English volition desire to encode their robots.txt paths using percent encoding. Percent encoding is safer because, although Google treats the two raw and encoded versions of the identical robots.txt regulation way equally, another crawlers could possibly construe raw characters differently.
Looking Ahead
According to the video description, the next Non-Canonical Cabinet event volition shield the 404 position code. The current Content-Usage outline is set to expire on Feb. 20, 2027, unless a new revision replaces it sooner.
Featured Image: Alicia97/Shutterstock
Google wordmark: Source: Google.
