Should I Block AI Crawlers Or Measure Their Value First? – Ask An SEO

Jul 02, 2026 04:30 PM - 1 week ago 16027

Today’s mobility looks beyond the emblematic traffic-driving goals of AI visibility to the worth those ample connection models supply a website owner, and asks:

“AI crawlers are visiting my website progressively often, but I can’t show whether they supply immoderate value. Should I let them, artifact them, aliases dainty different AI crawlers differently? How tin I measurement whether their activity leads to citations, referral traffic, aliases conversions earlier making that decision?”

Many SEOs don’t recognize the costs of having bots sojourn their site. Recently, pinch the proliferation of AI bots, the costs of allowing anyone and everyone to entree your contented are becoming an costly business.

Types Of AI Crawlers

First, let’s look astatine the different types of bots that sojourn a website.

Common bots that will beryllium visiting a website regularly see those we want to person entree to our site, for example, hunt motor bots. These aren’t the only bots, but they are often immoderate of the astir prolific consumers of bandwidth. Alongside hunt bots, location will beryllium tools. These tin see bots from uptime monitors, hunt and analytics tools, and information and vulnerability scanners.

Overall, website owners person to determine whether the bots visiting their tract should beryllium allowed to proceed aliases if they airs much harm than good. Examples of bots that tract managers often artifact are those that are trying to scrape merchandise accusation to provender different website’s database, aliases malicious bots looking for login vulnerabilities. Whether aliases not to artifact these bots is simply a reasonably easy determination – they airs a consequence to the intelligence spot of the marque aliases the information of the website.

AI bots mightiness really autumn location successful betwixt these “good” and “bad” bots.

AI Training Bots

These bots, for example, OpenAI’s GPTBot, are scouring the web for accusation to provender the AI training models. They are helping to create the knowledge guidelines that the LLMs are learning from, including entities and really they subordinate to each other.

For galore website owners, these are the astir arguable AI crawlers. Their superior intent is not to nonstop postulation backmost to your site, but to “read” and cod accusation that whitethorn beryllium utilized to train and amended models. In immoderate cases, that contented whitethorn later beryllium utilized to reply personification questions without generating a sojourn to the original source. This makes it harder to tie a nonstop statement betwixt the crawler’s activity and business value.

Search Indexing Bots

These bots, OpenAI’s OAI-SearchBot, for example, are reviewing pages and collecting accusation to aboveground and nexus websites successful LLM “search results,” not to train instauration models.

These are often easier to warrant allowing because their intent is person to that of a accepted hunt engine. If they are indexing your contented truthful that it tin beryllium cited successful AI-generated answers, they person a much evident way to creating visibility, referral traffic, and marque awareness.

User-Triggered Fetches

These bots, including OpenAI’s ChatGPT-User, retrieve pages connected request erstwhile users inquire astir circumstantial websites aliases documents, alternatively than relying solely connected a pre-built scale aliases knowledge base.

These fetches correspond genuine personification liking successful your site. They are specifically looking for further accusation aliases discourse connected your content, business, aliases products. This is simply a valuable parameter of their spot wrong the acquisition funnel. They person already discovered your marque and are now diving deeper into your content.

How To Block AI Bots

OpenAI updated its documentation truthful that ChatGPT-User, the user-triggered fetcher, nary longer commits to honoring a website’s robots.txt. Perplexity behaves successful a akin manner, pinch Perplexity-User. So the robots.txt, which SEOs person been reliably utilizing for years to power awesome bots, now only blocks the compliant training and hunt crawlers. For user-triggered and non-compliant bots, you request server aliases WAF-level blocking.

WAF-Level Blocking

A WAF (web exertion firewall) sits successful beforehand of a website’s server and acts arsenic an inspection checkpoint. A WAF tin beryllium configured to only let definite bots, aliases to let each but excluded bots. This is simply a very robust measurement of preventing unwanted bots from visiting a website.

Although this typically sits extracurricular the purview of an SEO, you whitethorn beryllium acquainted pinch immoderate of the brands that connection WAF-level blocking, for illustration Cloudflare and AWS. If you cognize which tech stack your website runs on, you whitethorn beryllium capable to investigation WAF blocking earlier presenting the thought to your infrastructure team. However, astir ample companies will already person a assortment of bots they are blocking, truthful endeavor teams will apt person a process successful spot for adding aliases removing bots from WAF lists.

Server Rules

Rules tin beryllium added straight to your server that analyse the postulation that is hitting it, and find if it comes from an unsafe bot. The server will cheque items for illustration whether the petition comes from a root utilizing automation aliases lacks the due headers. If it deems the user-agent arsenic unsafe based connected the rules, it will not fto the bot deed the site.

The Risk Of Blocking All AI Bots

This is wherever the dilemma lies. Some of the AI bots are scraping your website’s intelligence property. However, if you artifact them, that intends they whitethorn not aboveground your marque aliases products successful their answers, putting you astatine a competitory disadvantage.

The primary consequence pinch blocking AI bots is that you whitethorn find your tract nary longer cited successful LLM answers. Given the debased measurement of referral postulation LLMs are passing, that whitethorn look for illustration a consequence you are consenting to take.

However, what we do cognize is that, though LLMs aren’t passing the aforesaid measurement of postulation arsenic accepted hunt engines, they are adjuvant successful raising marque awareness. If your marque isn’t the 1 being cited, that intends a competitor’s is.

With everything AI-related, we person to retrieve that the section is evolving quickly. LLMs whitethorn not beryllium passing overmuch postulation correct now, but that doesn’t mean that will ever beryllium the case.

Preventing AI bots from crawling a tract now mightiness make the tract functionally invisible successful the early if LLMs go the superior find method.

In addition, blocking each AI bots removes your expertise to trial and learn. If you extremity each AI crawler from accessing your site, you suffer the opportunity to understand which platforms make visibility, which mention your contented accurately, and which person the imaginable to go meaningful postulation sources successful the future.

The Risk Of Allowing All AI Bots

There is, of course, a very existent threat that sites are facing from AI crawlers today, however. The 2 top risks travel from the ferocity astatine which the bots are crawling and consuming content.

Training On Intellectual Property

Many website owners are uncomfortable pinch the thought that proprietary contented aliases assets could beryllium utilized to amended an AI exemplary without immoderate nonstop compensation aliases attribution. This is 1 of the loudest complaints that we perceive from SEOs – you are visiting my site, taking my content, but I americium not getting postulation successful return.

The concern is peculiarly precocious for publishers and businesses whose competitory advantage comes from unsocial accusation aliases assets. If that contented becomes portion of a model’s training data, location is little request for users to sojourn the original website.

There is besides the consequence that bots whitethorn beryllium scraping information aliases contented that really forms portion of a merchandise aliases service. For an LLM to repackage that accusation and service it arsenic an reply aliases procreation tin beryllium devastating to businesses. For example, artists are seeing photos of their activity being ingested by LLMs and utilized to make images “in the style of” their ain creations. This usage of IP could beryllium straight impacting a business’s profits.

Crawl Costs

AI crawlers tin devour important server resources. Large sites often study AI bots requesting pages astatine a overmuch higher wave than accepted hunt motor crawlers.

This costs is not ever evident because it is often absorbed into wide hosting fees. However, astatine scale, excessive crawling tin summation bandwidth depletion and effect the acquisition of existent users if resources go constrained.

For immoderate organizations, the nonstop financial costs of serving AI crawlers is the superior facet down decisions to restrict aliases artifact them.

How To Identify Which Bots Are Visiting Your Site

The biggest blocker to knowing the consequence and reward to your marque from AI bots is knowing which bots are moreover crawling your site.

This information isn’t ever easy to travel by. Let’s spell done a mates of ways we tin place if a bot has aliases is crawling your site.

Log Files

Log files will beryllium the astir complete root of information connected which bots are visiting your website. Downloading a sample of logs from the past 30 days could springiness you a bully thought of what percent of your bots are linked to AI.

The log files will apt person each mode of bots successful them, and it mightiness return a spot of investigation to place which of them are AI crawlers. Once you person translated the user-agent accusation into thing much human-readable, it will beryllium a elemental lawsuit of adding up the hits of each bot and moving retired what percent of the full is from AI crawlers.

There are a batch of devices disposable that will automate this, however. There are a mates of types that mightiness thief pinch this workout – accepted log record analyzers and AI visibility search tools.

The log record analyzers will supply a breakdown of which bots are from accepted hunt engines, and which are from AI. The AI optimization tools, which are chiefly for search and analyzing your site’s visibility successful LLMs, often besides person an AI supplier search characteristic based connected your log files.

You should besides effort to understand whether circumstantial bots are concentrating connected peculiar sections of the site. A crawler many times accessing merchandise pages whitethorn bespeak that those assets are peculiarly valuable to the platform. This tin thief pass whether you let entree to the full tract aliases create much circumstantial restrictions.

See also: The Modern Guide To Robots.txt: How To Use It Avoiding The Pitfalls

Referral Traffic

If you don’t person entree to your log files, you tin still get an thought of which bots person visited your tract from the referral postulation they send.

Looking successful your analytics package astatine referral sources, you whitethorn admit a information arsenic LLMs, for illustration ChatGPT aliases Perplexity. Google Analytics has precocious deployed a new transmission classification called “AI Assistant.” This caller transmission makes it easier to spot what visitors person recovered your tract via an LLM, but it only recognizes ChatGPT, Gemini, and Claude via referrer header and does not seizure Perplexity. It is safe to presume that if an LLM has cited your website and provided a nexus for visitors to follow, its bot whitethorn person visited your tract astatine immoderate point.

This isn’t a foolproof method of seeing each the AI bots that person visited your site, because it will only uncover platforms that person sent referral postulation wrong the timeframe you are viewing. Any LLM bot that has crawled your tract but not sent referral postulation will stay chartless to you. It is besides imaginable that the citation that sent postulation to your tract came from training information aliases a cached type of your page. However, if you are genuinely incapable to entree log record data, this tin springiness you a adjacent approximation of the bots that person visited your website.

What Additional Data You Need

Beyond simply knowing if a bot has visited your site, it is basal to cognize the effect of their visit. This intends you request to find retired from the log files, aliases landing pages of their referred traffic, which pages the AI bots person crawled.

This accusation will springiness you a amended thought of wherever the bots are scraping information from, and whether they are pages you do aliases do not want them visiting.

Potentially the astir important constituent of information for this study is the costs of the AI bots hitting your site. This is apt accusation you will request to get from whoever manages your website server. They should beryllium capable to show you which bots are crawling the tract truthful overmuch they are already astatine the constituent wherever they are considering blocking them. This personification should besides beryllium capable to cipher really overmuch money it is costing your institution to let bots to crawl the site. This is very adjuvant accusation erstwhile it comes to the adjacent spot of the study – determining the worth of AI bots.

How To Measure Value

This adjacent measurement is captious successful the decision-making process. The mobility of whether to allow, block, aliases restrict an AI bot from your tract hinges connected the worth those bots provide.

Most website owners are alert that LLMs do not nonstop arsenic overmuch postulation to websites arsenic accepted hunt engines do. However, Cloudflare data from June 2025 suggests that for each 1 sojourn to a website, Anthropic’s Claude will person made 70,900 page requests, whereas for Google, that ratio is 9.4:1. This “crawl-to-refer” ratio is shockingly precocious for immoderate LLMs.

What Value Is The Traffic The LLMs Send?

The first measurement is knowing whether visitors arriving from LLMs are really valuable. Looking purely astatine convention numbers tin beryllium misleading. AI platforms presently nonstop importantly little postulation than accepted hunt engines, but the visitors they do nonstop whitethorn beryllium highly qualified.

Essentially, the cardinal measures to see present are engagement metrics. Are users from LLMs engaging positively pinch your tract successful a measurement that indicates they whitethorn go converting users? Even if they don’t acquisition thing connected their first visit, they whitethorn return via different transmission astatine a later date. Using your knowledge of personification journeys connected the site, comparison the behaviour of LLM-referred visitors pinch converting visitors from different channels.

Ultimately, the astir persuasive statement for allowing an AI crawler is gross procreation that outweighs the costs of them crawling the site. If visitors arriving from a circumstantial LLM spell connected to acquisition products aliases complete lead forms, they show they person affirmative business impact.

Citations And Mentions

Traffic is only 1 shape of value. A level that consistently cites your contented whitethorn beryllium expanding consciousness of your marque moreover erstwhile users do not click through. As SEOs, we cognize that postulation isn’t the be-all and end-all of marketing. Just because a visitant has not clicked to sojourn your website, it does not mean they will not jump successful their car to sojourn your brick-and-mortar shop they conscionable discovered done a Google Business Profile.

Consider LLMs successful a akin way.

Track really often your tract appears successful AI-generated answers for topics applicable to your business. The much often your contented is surfaced, the greater the likelihood that your marque is becoming associated pinch those topics successful users’ minds.

Sentiment

Being mentioned is not enough; knowing really your marque is being represented is arsenic important.

Review AI-generated answers to find whether your institution is being described accurately and positively. If a level often references your contented but misrepresents your products aliases expertise, that should shape portion of the decision-making process. An LLM that continually gets it incorrect is not conscionable costing your business successful server fees; it could beryllium costing your brand’s goodwill.

Query/Topic Coverage

Assess which topics, products, aliases services your marque appears for wrong AI platforms.

If competitors predominate important commercialized topics while your marque seldom appears, allowing applicable crawlers whitethorn go strategically important. Conversely, if you already person beardown visibility for cardinal subjects, you whitethorn beryllium much comfortable restricting definite types of crawlers.

Consider Future Value

One of the hardest aspects of this study is that today’s worth whitethorn not bespeak tomorrow’s value.

A crawler that generates small postulation coming whitethorn beryllium to a level that becomes a awesome find transmission successful the future. Equally, a crawler that appears costly coming whitethorn yet warrant its costs done improved visibility and referral traffic.

For this reason, debar evaluating AI crawlers solely connected short-term performance. Consider their imaginable strategical worth complete the adjacent respective years.

Build A Decision Matrix

The last portion of the study is simply a determination matrix. It’s a elemental measurement of organizing the AI crawlers into bots to “keep,” “restrict,” aliases “block.”

Using the accusation you person already gathered, inquire the pursuing bid of questions of each bot:

Does This Bot Provide My Site With Converting Revenue Or Useful Visibility?

Does this crawler lend to traffic, leads, revenue, aliases marque awareness? If it does, that is simply a beardown logic to support it. If it doesn’t look to supply immoderate postulation aliases visibility wrong the LLMs, past this is apt a “no” aliases “maybe.”

Is It Accessing Sensitive Information, Or Information We Want To Keep Proprietary?

This is wherever you analyse if it is safe to fto the bot roam freely, aliases if you person caught it scraping contented that is portion of your company’s IP. If that is the case, you will apt want to artifact it aliases restrict it.

How Trustworthy Is This Bot?

Is this a bot from a well-known AI company? Is location publically disposable archiving connected really its crawlers work, what commands they respect, and their information retention policies? If location is, this is simply a stronger motion that this is simply a bot that tin beryllium allowed to crawl your site. If location isn’t, past it is apt 1 to block.

Is This Bot Costing Us Significant Money Or Impacting User Access To Our Site?

This is simply a mobility astir the costs of letting the bot crawl your tract freely. If it is hitting the tract astatine a precocious frequency, it whitethorn good beryllium costing you a batch successful server fees. It could besides beryllium pushing the server past its capacity, which whitethorn forestall different adjuvant bots, aliases your existent tract users, from being capable to entree the site.

Can We Afford The Competitive Disadvantage From Not Allowing This Bot To Access Our Site?

This centers connected the consequence of your tract not being accessible to the bots.

If blocking a crawler would apt region your marque from a awesome AI platform’s answers, past the strategical costs whitethorn outweigh the infrastructure savings. If location is small grounds that the level references your contented aliases competitors, past the downside whitethorn beryllium limited.

The Final Decision

Once you person gathered each of your information and weighed up the pros and cons of each bot, you are fresh to make a decision. The cardinal to this decision-making is remembering that this whitethorn alteration complete time. You whitethorn not request to artifact a bot today, but you whitethorn want to restrict it for now, knowing you tin artifact it wholly astatine a later date.

Keep – Doesn’t Cost Much/Brings In More Value Than It Costs

These are bots that supply measurable value. This whitethorn beryllium done traffic, citations, marque visibility, aliases early strategical importance, but importantly, this worth outweighs the operational burden.

Monitor Or Restrict – Doesn’t Have Much Value But Doesn’t Cost Much

These are bots wherever the business lawsuit remains unclear. You whitethorn take to limit crawl rates, restrict entree to circumstantial areas of the site, aliases proceed gathering information earlier making a last decision.

Block – Low Value/High Risk

These are bots that create important costs, entree delicate content, aliases supply small grounds of existent aliases early value.

See also: WordPress Robots.txt: What Should You Include?

Going Forward

A cardinal constituent to retrieve is that this is not a lawsuit of “set it and hide it.” New AI bots will beryllium created. Bots that you person blocked whitethorn summation successful imaginable worth complete the adjacent fewer months and years.

As portion of your appraisal you request to build successful regular reviews. These mightiness beryllium triggered by the personification who is responsible for server costs asking you if you really request ChatGPT to beryllium accessing the site. Ideally, though, it will beryllium thing that you are proactively considering and that you tin coming to your stakeholders arsenic some a marque protection and future-proofing plan.

Consider reviewing your artifact database erstwhile a quarter. This is simply a cadence that doesn’t put excessively overmuch unit connected the personification pulling the log files, and besides gives you clip to make strategical changes if needed.

The cardinal takeaway is that location is seldom a bully logic to either let each AI crawler aliases artifact them all. Instead, dainty each bot arsenic an individual business case. Measure its cost, measure the visibility it provides, understand the consequence it creates, and past make a deliberate decision. That attack is acold much apt to protect some your existent resources and your early discoverability.

More Resources: