Why would an AI hunt provider have a long-term incentive to rely data that a brand deliberately keeps distinct from its public-facing website?
This is a inquiry I’ve been coming rear to as additional conversations concerning optimizing for AI hunt appear to affect creating satisfied specifically for agents, fairly than people.
The logic is understandable: AI systems need information, sometimes in formats or structures that aren’t particularly helpful to a individual visitor. If we can create that data easier for agents to consume, certainly that’s a fine thing?
Yes, but there is a important difference between making data additional accessible to AI and creating an substitute type of your website, or equal your brand, that exists chiefly to power what AI systems say concerning you.
We’ve spent decades dealing alongside versions of this issue in SEO, and I’m not convinced we’re taking those lessons earnestly enough.
How Can SEOs See This Coming?
Anyone who has worked in SEO for lengthy adequate volition have seen hunt engines develop in reply to the method used to manipulate them.
Early SEO was complete of practices designed to power hunt engines fairly than assistance users: hidden text, keyword stuffing, entrance pages, cloaking, and eventually increasingly advanced satisfied and nexus manipulation.
Search engines didn’t create rules against these practices since they disliked SEOs (mostly). They did so since allowing websites to falsify relevance and authority undermined the norm of their results.
Google’s former gives us a fairly apparent image of this evolution:
- 2011: Panda targeted low-quality content.
- 2012: Penguin tackled nexus spam.
- 2022: Google’s Helpful Content scheme sought to improved reward satisfied created to assistance people, fairly than chiefly diagram hunt traffic.
- 2024: Google’s updated spam policies made the stance on scaled satisfied abuse clearer, irrespective of whether the satisfied was created by AI or humans.
This is a extremely condensed type of things, and the actuality was far additional complicated, but the form is important.
As hunt engines became additional sophisticated, so did the methods used to power (spam) them. Search engines responded by investing additional attempt into deciding what data was trusted and rewarded.
Google motionless explicitly identifies cloaking as spam. I am not saying that cloaking’s definition is as uncomplicated (or east to detect) as that, but the parallels are too near to ignore.
So What Are We Building For AI?
We’re now seeing a gathering of approaches designed to create data accessible to AI systems and agents.
The llms.txt proposal is one example, providing a organized way of directing tongue models towards helpful website content. There are additionally agent-specific markdown pages, dedicated data endpoints, organized feeds and increasingly advanced methods of supplying data immediately to agents.
Cloudflare’s Markdown for Agents is a fine example of serving an substitute type of existing content. The data begins from the identical page, but is presented in a format that’s possibly easier for agents to process.
That’s basically distinct from creating new data that exists lone for AI consumption.
Making the identical data accessible in distinct formats isn’t the problem.
Making distinct data accessible to distinct audiences is anywhere things rotate into considerably additional complicated.
I can comprehend why brands power be attracted to this as getting satisfied onto a website has never been particularly easy. Content ownership, CMS limitations, scheme requirements, betterment resources, governance, approvals and the competing priorities of distinct teams all perform a part.
If you can bypass all of that by publishing a content file, or hosting a gathering of AI-specific satisfied location else, you have an appealing shortcut.
The Problem Of Trust And The Risk/Reward Here
There’s another crucial difference between traditional hunt and AI search: Google isn’t the lone important performer here. As I’ve explored previously, digging into ChatGPT’s hunt sources, there isn’t a single, accordant retrieval procedure underpinning all AI-generated answer as well.
Different systems, hunt providers, tools, and retrieval pipelines can all perform a function in what data is discovered and ultimately used.
Some of these processes inherit established “search” infrastructure, including the systems those that measure the relevance and reliability of information. But beyond Google or Bing, another sources behave differently.
That doesn’t average ChatGPT or another AI systems automatically rely everything they’re given. But there is a significant difference between having mechanisms to acknowledge dependable sources and having the cognition that traditional hunt engines have dealing alongside group attempting to manipulate those mechanisms.
We additionally need to distinct the capability to recover data from the decision to rely it. When I looked at how inquisitive agents really are, I saw how they use the smallest attempt to retrieve, summarize, and move on. Just since the correct answer was accessible doesn’t average it would continually be established and used.
If an delegate encounters data created specifically to power its answer:
- How should it established whether that data is accurate, authoritative, or accordant alongside what the brand tells its genuine customers?
- What happens whenever the machine-readable type says one thing, but the community website says item else?
- Who decides which is additional trustworthy?
- And, perchance most importantly, why should AI providers continue to obtain data that is deliberately divided from what users can independently verify?
I don’t think we have answers to these questions yet.
Haven’t We Been Here Before?
I don’t think it is fair or fact to assertion that all AI-specific satisfied format is a form of cloaking, or that making data accessible to agents is manipulative. Structured data, XML sitemaps, APIs, and merchandise feeds have been part of the web for years. There are entirely lawful reasons to provision machines data in distinct formats – in fact, I’d importune you do it!
But there’s a difference between assisting machines comprehend your endeavor and creating satisfied fair to power AI output. Especially whenever that satisfied isn’t represented anyplace group can readily access.
The incentive to manipulate is obvious, and whilst AI hunt isn’t traditional search, the incentives of AI hunt providers are opposed to this. They need to come back useful, exact answers and keep person trust. Why would they desire to advance an ecosystem of increasingly fragmented, difficult-to-verify sources that be chiefly to power their answers?
Perhaps these systems volition rotate into improved at validating data against another sources. Maybe agent-specific data volition be treated alongside greater scrutiny, or its power constricted without autonomous corroboration.
We don’t cognize exactly how this volition evolve, but assuming that today’s emerging practices volition remain acceptable, effective, or equal helpful in the lengthy term seems worryingly optimistic or naive.
Getting The Information Customers Need Onto A Website Has Always Presented Challenges
We’re “solving” an old issue alongside a new shortcut, and if it is too fine to be true, it may fine be!
For years, satisfied strategies have been constrained by what businesses desire to publish and what their inner processes authorize them to maintain. Often this creates a gap between the data customers desire and the data brands create available.
Generative AI has made this additional apparent and has been a important multiplier here! People can now ask detailed, particular questions concerning products, services, properties, experiences, and practically item else, expecting helpful answers without having to navigate a website to part the data together themselves. That creates a much greater petition for clear, accessible, and particular information.
If that data isn’t available, AI systems may have to appearance elsewhere or merely neglect to answer the inquiry correctly. So we do need to reconsider the data we’re publishing, how it’s organized and how distinct systems admission it.
But why should the resolution be to create another satisfied estate? One that requires distinct maintenance, governance, possibly another tech stack, and additional processes to justify it remains accordant alongside the data customers can really see?
Most battle to keep one type of their website exact and useful… I’m not certain creating a second, small apparent one is the apparent solution.
Useful For AI, Useful For People
I don’t think the answer is to refuse agent-specific formats or importune that AI systems should consume websites in exactly the identical way as people. That would disregard the genuine differences in how these systems operate, and the opportunities they create.
Equally, assuming the resolution is merely to create satisfied specifically for AI is avoiding a issue we’ve been trying to resolve for years.
The goal have to be to create genuinely helpful data accessible to both, from a consistent, authoritative origin that can be maintained, validated, and, anywhere appropriate, independently verified.
Information that’s helpful lone to AI creates a governance issue and raises questions of trust.
Information that’s helpful lone to group increasingly creates a visibility problem.
There is a sensible center dirt here, and it doesn’t necessitate us to invent another type of the web. I’ve written previously concerning why Google’s reluctance to endorse an entirely distinct set of AI optimization practices makes sense. There’s small incentive for hunt providers to legitimize another industry of techniques they volition afterward need to police.
The additional we advance brands to create data specifically for agents, detached from the public-facing web, the additional we hazard recreating problems that hunt engines have spent decades trying to solve.
I’m not convinced that building a shade web for AI is progress. It may fair be another shortcut we’ll expend the next decade trying to undo.
More Resources:
- Google’s Mueller Says llms.txt Can’t Help LLMs Differentiate Sites
- A Little Clarity On SEO, GEO, And AEO
- Google Answers Question About SEO For AI Agents
This article was initially published on Chris Green SEO.
Featured Image: DC Studio/Shutterstock
