Google’s Mueller Shares Their Experience With Markdown For AI SEO

Someone connected Reddit asked if anyone was having occurrence providing markdown files to LLMs. Google’s John Mueller shared his individual acquisition testing markdown files for LLM consumption.

Markdown Files For LLMs

Markdown files are instrumentality and human-readable contented that contains markup that signals whether thing is simply a header and truthful on. It’s fundamentally conscionable contented pinch each of the interactivity removed, truthful nary JavaScript aliases CSS.

The thought is that serving these files to LLMs will alteration them to devour the contented without having to woody pinch the extraneous HTML, JavaScript and everything other that is location specifically for quality interaction.

The information for serving markdown files to LLM is to get an advantage successful ranking successful AI hunt surfaces.

Has Anyone Had Success With Markdown Files?

A personification connected Reddit started the chat disconnected pinch this question:

“I already cache my html web pages truthful its not a immense magnitude of activity converting them to markdown and caching them again, and past checking the headers to spot if a bot asks for markdown. I’ve publication each the worldly I tin find online and the statement seems to beryllium that it’s not going to thief pinch getting much AI citations, but it won’t do immoderate harm either.

I’m wondering if it’s still worthy doing anyway, purely to extremity AI bots hammering my site. I had to artifact a fewer successful Cloudflare because they were going nuts, but if they’re requesting smaller markdown files past possibly it will beryllium worthy allowing them backmost successful again (I tin trim the files down to astir a 3rd of the size).

I suppose my mobility is, has anyone really seen 1 of the awesome AI bots petition markdown?”

Google’s John Mueller shared his acquisition that the only bots hitting his markdown files are from SEO tools.

Mueller shared:

“On my trial sites the only crawlers who declare to judge markdown are SEO tools. Ymmv.

(Also, it’s annoying that the server setups that I usage don’t log the judge header, truthful you person to manually group things up for logging moreover if you conscionable want to spot if anyone would judge it. If you’re funny astir your sites, I’d urge figuring retired really to log that and checking the metrics first.)”

Markdown Files Are Like Keyword Meta Tags

The problem pinch Markdown files for LLMs is that each the generative hunt and LLM companies person afloat mastered crawling and indexing HTML files. Further, it’s successful their liking to download what users see, not typical contented that’s made conscionable for LLMs.

The logic is because tract owners and SEOs can’t beryllium trusted pinch contented that’s conscionable for LLMs. This is the logic why the keyword meta tags failed. Everyone stuffed it pinch keyword variations, misspellings, and phrases that did not look connected their content.

There is nary upside for LLMs and AI agents to devour markdown files because they aren’t trustworthy and because downloading and indexing HTML contented is trivial, it’s a problem that’s been solved for good complete 30 years.

Markdown For AI Agents

Cloudflare is nether the belief that markdown files are wildly celebrated pinch LLMs. They connection services that tin supply markdown files to AI agents that fundamentally inquire for it.

They precocious wrote:

“Markdown has quickly go the lingua franca for agents and AI systems arsenic a whole. The format’s definitive building makes it perfect for AI processing, yet resulting successful amended results while minimizing token waste.

Cloudflare’s web supports real-time contented conversion astatine the source, for enabled zones utilizing contented speech ↗ headers. When AI systems petition pages from immoderate website that uses Cloudflare and has Markdown for Agents enabled, they tin definitive the penchant for text/markdown successful the petition and our web will automatically and efficiently person the HTML to Markdown, erstwhile possible, connected the fly.”

Honestly, that’s conscionable not reality. If it were that Reddit chat would person been filled pinch markdown record occurrence stories.

Instructions For AI Agents

OpenAI and Anthropic are much realistic astir markdown files because that’s the “lingua franca” for AI supplier skills, guidance, and instructions.

An OpenAI support page explains:

“Custom instructions pinch AGENTS.md
Give Codex other instructions and discourse for your project

Codex sounds AGENTS.md files earlier doing immoderate work. By layering world guidance pinch project-specific overrides, you tin commencement each task pinch accordant expectations, nary matter which repository you open.”

Takeaways

• There is nary grounds present that serving Markdown versions of webpages improves AI citations aliases AI hunt visibility.
• Google’s John Mueller says the only crawlers connected his trial sites that nosied astir his markdown files were SEO tools.
• Mueller’s proposal is to group up ways to way whether AI bots are really requesting Markdown earlier going all-in generating them for bots.
• All AI systems tin already crawl and scale HTML. There is nary request connected their extremity for a abstracted Markdown type of HTML pages.
• Cloudflare supports automatic conversion of HTML to Markdown erstwhile AI systems petition it. But their deptiction of AI supplier request for it is overenthusiastic.
• Markdown is already useful for AI supplier instructions and context, including OpenAI Codex’s usage of AGENTS.md arsenic good arsenic for Anthropic Claude.
• Markdown’s usefulness for AI agents does not widen to providing advantages for crawling, indexing, and ranking successful AI hunt engines.

Featured Image by Shutterstock/ezphoto