Google: Cloudflare Content Signals Robots.txt Directive Has No Effects

John Mueller from Google said that the caller Content Signals robots.txt directive invented by Cloudflare last year has "no effects whatsoever for immoderate crawler aliases LLM." He added that it "just adds bloat and early attraction to your robots.txt file." As acold arsenic he knows, he said, nary of the crawlers aliases LLMs usage the content-signal robots.txt directives.

This was a consequence that John Mueller from connected Reddit successful consequence to the question, "Testing whether Content-Signal headers and llms.txt really thief pinch Person entity disambiguation."

Here is what John Mueller posted:

* Google doesn't usage llms.txt aliases llms-author.txt. I don't cognize of immoderate different crawler / llm confirming they're utilizing these (other than SEO tools).

* AFAIK nary of the crawlers / llms usage the "content-signal" robots.txt directives. It was made up by a CDN, afaik it has nary effects whatsoever for immoderate crawler aliases llm. Using it conscionable adds bloat & early attraction to your robots.txt file. You tin besides adhd different arbitrary things to your robots.txt file, crawlers conscionable usage the directives that they support and disregard the rest.

As you whitethorn person heard, Cloudflare group a deadline of September 15, 2026 wherever it will beryllium mounting caller defaults for each of these 3 classifications. For each caller domains onboarding to Cloudflare, the categories of Training and Agent will beryllium blocked by default connected the pages that show ads, while Search will stay allowed by default.

Cloudflare is utilized by astir 21.3% of each websites connected the Internet arsenic of January 2026. So if you usage Cloudflare, you should make judge you want this to happen. I already checked my settings successful Cloudflare earlier that deadline.

That being said, John Mueller is intelligibly saying that Google has not yet respected the Content Signals robots.txt directive and he makes it sound for illustration the hunt institution has nary plans connected doing so.

Forum chat astatine Reddit.