Indexed Claude Chats Show Why Disallow Is Not Noindex

Over the weekend, shared Claude chats unexpectedly showed up successful Google hunt results.

On July 27, I checked retired the crawl rules connected the Claude stock pages and recovered a noindex directive that was hidden down a block, preventing Google from accessing it

The stock way is blocked by robots.txt connected claude.ai. These pages besides nonstop backmost an X-Robots-Tag header group to none, which Google’s robots meta tag guidelines dainty arsenic the aforesaid arsenic noindex and nofollow.

Because of this, the rules conflict pinch each other. According to Google’s guidance, a noindex tag only useful if the crawler is allowed to entree and publication the page. If a page is blocked by robots.txt, it tin still beryllium indexed if different pages nexus to it, since Googlebot notes the URL without really opening it.

The pairing is not unsocial to AI companies. If Google discovers a blocked URL elsewhere, it tin database the URL without crawling the noindex.

What Happened

404 Media reported connected Monday that Claude stock pages were showing up done the site: operator. TechCrunch reported being capable to find aesculapian material, soul institution documents, and files carrying the names and telephone numbers of superior school-aged children. Artifacts, the documents and mini apps built wrong Claude, showed up done a 2nd path.

Much of the coverage, including The Decoder, reported that the stock pages lacked a noindex tag. As of July 27, a noindex header is being served and sits down a robots.txt block. It’s unclear whether that header was successful spot earlier the weekend, arsenic I couldn’t corroborate this.

What The Crawl Rules Return

Findings from my July 27 checks against claude.ai:

• The robots.txt record astatine claude.ai disallows /share/* nether the User-agent: * group, pinch nary abstracted Googlebot group, truthful that is the group Google goes by.
• A unrecorded stock URL responds pinch the header x-robots-tag: nary connected a GET request.
• The aforesaid header appears erstwhile the petition claims to beryllium Googlebot, and the consequence includes Vary: User-Agent.
• The /public/artifacts/ way is not listed successful robots.txt.

I didn’t verify whether artifact URLs return a page-level noindex, truthful these 2 paths can’t beryllium compared based connected their controls.

Independent IT advisor Daniel J. Glover reported the aforesaid conflict connected July 26. My cheque shows the setup still successful spot connected July 27, aft it was reported that the chats were nary longer displayed successful hunt results, but I couldn’t found what was served to Google erstwhile those URLs were first discovered.

Why Blocking A Page Doesn’t Remove It

Robots.txt helps guideline really hunt engines crawl your website, but it doesn’t determine what gets indexed. Google has been clarifying this for years.

For example, John Mueller explains that moreover pages blocked by robots.txt tin look successful hunt results if different pages nexus to them. If you want to support a page retired of hunt results, utilizing the “noindex” tag is the correct choice. Martin Splitt besides recommends avoiding placing some rules connected the aforesaid page. Google precocious emphasized this favoritism again, arsenic seen successful their update past week.

Blocking URLs pinch robots.txt doesn’t region already indexed pages. For a much imperishable removal, Google’s guidance is to usage a “noindex” tag that its crawler tin read.

Anthropic’s Response

Anthropic told TechCrunch that stock links will look successful hunt results only if group station them successful accessible locations for crawlers, and that links sent done backstage messages won’t beryllium indexed.

Spokeswoman Amie Rotherham added that these links “are not guessable aliases discoverable unless group take to stock them themselves.” This addresses 1 measurement a nationalist stock URL tin beryllium discovered, but it doesn’t reside the conflicting rules I recovered connected July 27.

Google spokesperson Ned Adriance stated that hunt engines do not find which pages are made public, and that Google provides tract owners pinch power complete crawling and indexing and follows those controls.

Why This Matters

Any chats aliases artifacts you haven’t unshared are still accessible to anyone pinch the link. In Claude, spell to Settings > Privacy > Shared Chats to spot a complete list.

Unsharing a chat disables the nonstop link, but removing the URL from Google is simply a different thing. As a Claude user, you can’t power what it serves to Googlebot aliases taxable a removal request, arsenic these are managed by Anthropic.

Artifacts are published done their ain process, which makes unsharing a chat and unpublishing an artifact from that chat 2 abstracted actions.

Beyond settings changes, see establishing a norm astir sharing AI chats and artifacts. If a chat contains thing you wouldn’t station publicly, don’t stock it done a nexus that mightiness get indexed later.

Looking Ahead

The shape isn’t unsocial to Anthropic. OpenAI pulled shared ChatGPT chats from hunt successful August 2025, and Google blocked Bard transcripts from indexing successful 2023. Each lawsuit progressive a nationalist stock URL that traveled further than its proprietor whitethorn person expected.

Three companies person now had customer chats move up successful search. That’s logic capable to dainty a nationalist stock nexus arsenic a nationalist webpage from the infinitesimal it exists.

Featured Image: Cast of Thousands/Shutterstock