97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows

Ahrefs analyzed logs from 137,000 domains and recovered 97% of llms.txt files sewage zero requests. No bots, nary humans.

The analysis utilized Ahrefs information to place personification agents fetching files. Around 28% of 137,000 domains people an llms.txt file, but since Ahrefs’ customers are much technical, the existent take connected the broader web is apt lower.

Of astir 38,000 domains pinch valid files, only astir 1,100 received immoderate traffic.

Of files pinch requests, 96% came from bots, mostly non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.

Who Fetches llms.txt Files

SEO audit devices had 21% requests, past unidentified bots (14%), web crawlers for illustration Googlebot (13%), and tech profiling devices for illustration BuiltWith (11%).

AI bots, crossed 4 categories, made up 19% of requests. AI is the largest segment, but the breakdown differs from astir llms.txt advocates’ expectations.

Coding agents sent 10% of requests, training crawlers 5%, assistants 2%. Claude-Code and GPTBot were the apical individual bots.

Slackbot unsocial fetched llms.txt files much often than PerplexityBot did.

The Industry Studying Itself

The study recovered 12% of requests from devices that audit, scan, aliases study llms.txt files alternatively than usage them.

GEO and AEO readiness devices sent 5% of requests; dedicated scanners and validators sent 3%, much than AI retrieval bots and assistants combined. Research bots sent 2%, pinch the largest investigation crawler identifying arsenic a punctual injection survey.

An ecosystem has developed astir scoring and cataloging a record format earlier a important assemblage appears.

No AI Bot Looks For Files That Don’t Exist

Requests for /llms.txt paths pinch 404 errors drew nary AI traffic. Humans hitting those 404s look to beryllium group typing the URL into browsers, apt checking competitors.

The Chrome Lighthouse llms.txt audit, which reignited the llms.txt statement successful May, generated astir 22 requests crossed the dataset, astir 1 successful 1,000.

Why This Matters

The information lines up pinch what Google’s John Mueller has said astir llms.txt for complete a year. Lily Ray pressed Mueller connected the spread betwixt Google Search’s dismissal and Chrome’s Lighthouse audit. He said llms.txt is “not done for search” and called it a “temporary crutch, possibly to prevention immoderate tokens” for AI coding tools.

The information shows the file’s assemblage is coding agents and training crawlers, not AI hunt and retrieval bots that would make citations.

We reported connected the divided betwixt Google Search and Lighthouse archiving successful May. SE Ranking’s earlier analysis of 300,000 domains showed nary relationship betwixt having llms.txt and AI citation frequency. Ahrefs’ information points to 1 imaginable reason: the bots astir straight tied to unrecorded AI retrieval hardly requested these files successful May.

Looking Ahead

The punctual injection uncovering is worthy watching. Ahrefs recovered a crawler studying llms.txt arsenic a punctual injection risk, since agents spot ingested content. Sites auto-generating these files via CMS should reappraisal their content.

Every fig successful this study is simply a ceiling. Ahrefs measured requests, not whether bots acted connected what they fetched.

Featured Image: sdecoret/Shutterstock