Google On Using Markdown For AI SEO

Google’s John Mueller responded to a station connected Bluesky that lamented each the effort being wasted connected AI supplier accessibility alternatively of focusing connected the much productive activity of making sites accessible for humans.

Tactical SEO

One of the accordant aspects of the history of SEO is that practitioners thin to travel tactical trends if it’s evident that everyone other is doing it. I conjecture it’s quality nature.

Back successful the early days location was a feline who built a directory and purchased capable precocious PageRank links to the location page to get it to astir a PageRank of 7 (on a standard of 1-10). SEOs were practically pushing each different retired of the measurement to manus this feline their money. He sold thousands of links from his directory, and cipher thought to cheque if immoderate of the links really made a ranking difference. It was easy to check, but cipher did.

The latest maneuver is LLMs.txt and creating markdown pages for AI supplier consumption. Cloudflare announced successful February that their infrastructure tin automatically create markdown files, mostly arsenic a measurement for developers to prevention LLM token consumption. Their announcement stated that OAI-SearchBot, OpenAI’s hunt crawler, was consuming markdown pages.

OpenAI’s charismatic documentation explicitly positions its OAI-SearchBot arsenic crawling websites and does not mention aliases urge markdown files.

“OAI-SearchBot is for search. OAI-SearchBot is utilized to aboveground websites successful hunt results successful ChatGPT’s hunt features. Sites that are opted retired of OAI-SearchBot will not beryllium shown successful ChatGPT hunt answers, though tin still look arsenic navigational links. To thief guarantee your tract appears successful hunt results, we urge allowing OAI-SearchBot successful your site’s robots.txt record and allowing requests from our published IP ranges below.”

Yes, it tin crawl markdown if it’s pointed to it but that’s not what it’s retired location for.

Stephanie Walter posted:

“Sad truth: we are making the web accessible for AIs, not for people.
Some sites now connection a matter type for LLMs, but still skip existent accessibility needs for illustration due heading structure, landmarks that surface scholar users need.”

Google’s John Mueller responded:

“A decently made website useful good for AI agents … and hunt engines, and LLMs, and supra all, for existent people.

If you’re trying to hole accessibility issues by making a abstracted “agent-friendly” version, you are conscionable building method debt. You’ll person to redo it aggregate times. Just hole it.”

AI Agents Crawl HTML

It’s a comparatively trivial point to crawl HTML. Search engines person been doing it for complete 30 years. Making sites accessible for everyone makes sense, particularly since it’s personification behaviour signals that person ever driven hunt motor rankings. From links to users searching pinch a marque name, Google has consistently utilized personification signals for ranking purposes.

So, from the position of SEO, it makes a load of consciousness to make websites accessible for everyone.

Featured Image by Shutterstock/DETHAL