Llms.txt V2 Adds Formal Markdown Linking For AI Agents

Jeremy Howard, the Answer.AI developer who created llms.txt, published a version 2 update to the spec connected August 10. It’s the first revision since the format launched, and it adds modular ways for agents to observe Markdown versions of pages.

The 2024 connection included the llms.txt record and Markdown versions of individual pages, which are published astatine the aforesaid URL pinch .md added to the filename. V2 keeps this but introduces a 2nd URL shape for Markdown pages and general ways for agents to find them. For sites already utilizing the file, updating it to V2 requires only a insignificant alteration alternatively than a complete rebuild.

What Changed In V2

V2 now supports 2 URL patterns for Markdown pages. Sites tin either adhd .md to the afloat filename, which was the only shape supported before, making /docs/tutorial.html go /docs/tutorial.html.md, aliases they tin switch the hold altogether, turning the aforesaid page into /docs/tutorial.md.

To make it easier for agents to observe Markdown pages, the V2 update introduces 2 nexus relations. A rel=”alternate” property pinch type=”text/markdown” provides a nonstop nexus from a page to its Markdown version. Additionally, a rel=”describedby” property points to the llms.txt record that covers it, because a azygous llms.txt record tin picture each pages nether its path.

Sites tin see nexus relations either arsenic modular HTML nexus elements wrong the page caput aliases done an HTTP Link consequence header. The header action is versatile, arsenic it tin beryllium utilized connected files that aren’t moreover HTML, for illustration Markdown pages. Plus, it tin beryllium added via server aliases CDN setup, truthful there’s nary request to alteration a site’s templates aliases code.

Why Howard Made The Change

Howard said the update is based connected 2 years of applicable experience. Now, thousands of sites are publishing an llms.txt file, and platforms for illustration Mintlify thief by creating 1 for each tract they host. Google’s Chrome squad has moreover added a cheque for this record to Lighthouse’s Agentic Browsing category, which we discussed erstwhile it was introduced successful May. Additionally, Anthropic, OpenAI, and Google’s Gemini team now people their ain llms.txt files for their developer documentation.

As take grew, a existent spread showed up. The llms.txt record pointed agents toward pages, but thing successful the spec told them wherever the Markdown versions really lived.

Where This Leaves Google’s Guidance

The update doesn’t alteration the positions of Google’s 2 merchandise teams. Google’s Search squad has said for complete a twelvemonth that support for llms.txt isn’t connected their agenda, and the company’s AI optimization guide states that Google Search itself doesn’t utilize these files. The guideline besides notes that maintaining 1 of these files “will neither harm nor thief your site’s visibility aliases rankings” because Google Search ignores them.

Chrome’s Lighthouse instrumentality has an Agentic Browsing check that attempts to retrieve the llms.txt record and flags server errors. A missing record that returns a 404 is treated arsenic N/A instead, because providing the record is optional. The instrumentality views this arsenic a measurement to thief browser agents publication a site’s layout much quickly. However, it doesn’t specifically trial for caller nexus relations aliases Markdown-page discovery.

Why This Matters

The update to V2 involves adding 2 nexus relations arsenic HTML nexus elements aliases utilizing 1 HTTP header. What’s really important is what this update indicates astir who the format is designed to serve.

Coding agents and archive devices are already reference llms.txt files to thief velocity up API lookups. This usage lawsuit isn’t related to whether Google uses the record arsenic a ranking input. V2 clarifies this behaviour alternatively than making a caller lawsuit for hunt visibility.

Websites that support an llms.txt record chiefly to power AI Overviews aliases AI Mode citations still person nary information aliases Google connection backing that goal.

Those who support it because coding agents, IDE tools, aliases archiving platforms already usage it now person a clearer spec to reference and a defined measurement to nonstop devices to Markdown contented erstwhile available.

Looking Ahead

Howard’s changes page shows each summation to the llms.txt spec on pinch the reasoning down it. V2 is still unfastened for feedback connected GitHub, truthful if you’re utilizing the nexus relations now, retrieve that the syntax mightiness still alteration earlier everything is finalized arsenic a unchangeable version.

Featured Image: EyeFound/Shutterstock