Should I Block AI Crawlers At Robots.txt Or Server Level? – Ask An SEO

Sep 03, 2026 07:00 PM - 1 hour ago 3

Deciding to artifact AI crawlers is simply a business determination that galore hunt professionals are presently discussing. But erstwhile you’ve made the decision, what’s the champion measurement to spell astir blocking those bots?

There are 2 main approaches to blocking crawlers to consider: done robots.txt and astatine the server stack.

The Two Approaches

Both of these approaches person their pros and cons. Let’s commencement by examining really they activity and the differences betwixt the two.

Blocking Through The Robots.txt

Blocking AI crawlers utilizing robots.txt is precisely the aforesaid process arsenic you would usage for blocking immoderate type of bot.

Each AI bot has its ain identifying name, for example, OpenAI’s GPTBot and OAI-SearchBot. To artifact them, you simply request to adhd a disallow rule specifying the crawler’s name. For example, to forestall GPTBot from crawling immoderate portion of your website, you would add:

User-agent: GPTBot

Disallow: /

If location are only definite parts of your website you want to forestall the AI bots from crawling, you tin telephone those retired successful the aforesaid way. For example, to forestall GPTBot from crawling your merchandise pages you would see the files those pages beryllium in, e.g.:

User-agent: GPTBot

Disallow: /products/

Blocking At The Server Level

There are a fewer ways you tin artifact bots astatine a server level: done the server itself, the CDN aliases the WAF.

In this instance, the server will publication the incoming request, for illustration the bot’s IP, header, etc., and use the circumstantial rules you person configured for that supplier (deny, allow, redirect). For example, you tin specify that GPTBot receives a “deny” command. This would forestall the bot from accessing the contented connected your site.

For Content Delivery Network (CDN), the conception is the aforesaid but it happens astatine an earlier shape of a bot’s visit. The CDN intercepts a petition for contented from a bot earlier it hits the server. This fundamentally saves server bandwidth arsenic the bot ne'er really interacts pinch it. Some CDNs connection this exertion natively without you having to do overmuch to configure it. For example, Cloudflare offers preset blocking based connected whether a bot is simply a hunt crawler, an supplier aliases utilized for training, arsenic good arsenic allowing finer-tuning connected a bot by bot basis.

At the Web Application Firewall (WAF), bots are scrutinized much than the CDN does. The WAF acts arsenic a information furniture that tin analyse petition behavior, not conscionable the headers utilized by the bots. This intends it is tin of detecting bots that are spoofing different user-agents. It is the astir competent measurement successful astir tech stacks of identifying much blase AI crawlers that are looking to gaffe nether the radar of blocking attempts. The WAF your institution is utilizing whitethorn beryllium portion of your CDN, for example, Cloudflare WAF, aliases a standalone exertion for illustration AWS WAF.

Robots.txt: Pros And Cons

The robots.txt is perchance the astir accessible measurement for hunt professionals to power bots. Typically, SEOs person entree to change the robots.txt for their domains, aliases tin easy petition a speedy update by the improvement team.

However, location are immoderate different benefits to utilizing this method.

Pros

The robots.txt disallow system is officially supported by the largest, reputable AI companies. For example, OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot, Claude-User and Claude-SearchBot, Google’s Google-Extended, and Perplexity’s PerplexityBot.

This method allows you to selectively take which pages to forestall the bots from visiting, and besides to fine-tune the blocking based connected each crawler.

Cons

There are immoderate cons to this method, however. The top consequence is that compliance pinch the robots.txt is wholly voluntary and not centrally monitored. That is, though AI bot creators whitethorn declare their bots respect the robots.txt, it is conscionable a group of requests, not an existent block. Think of it arsenic a no-trespassing motion successful beforehand of an unfastened gate. There is thing really stopping the bots, only their being coded to respect the rules of the robots.txt.

The robots.txt tin beryllium configured to disallow bots from definite pages very easy if location are robots.txt controls successful the website’s CMS. This intends that non-technical stakeholders tin accidentally artifact much bots than anticipated pinch a mistaken disallow rule. This tin beryllium catastrophic if the robots.txt is updated to disallow each bots, for example, by implementing:

User-agent: *

Disallow: /

The robots.txt isn’t automatically updated erstwhile new personification agents are released. This intends that personification will request to manually adhd caller disallows whenever you want to forestall a caller AI bot from accessing your site.

Server Stack: Pros And Cons

Blocking bots astatine a server, CDN, aliases WAF level has different pros depending connected the implementation.

Pros

The CDN and WAF implementations will extremity bot requests earlier they deed the server. This will prevention server bandwidth, reducing the strain connected the server and redeeming associated costs.

The biggest pro for the server stack implementations, nary matter which you choose, is that they are a definite block. If the robots.txt is simply a polite “no trespassing” sign, the server, CDN, and WAF blocks are a padlock connected the gate. These implementation methods do not require a crawler’s compliance; they observe the bots and extremity them from accessing content, whether the bot is compliant aliases not.

Another use of this method is that the package that sits astatine these levels will often springiness reports connected the bots that person been blocked. The “padlock” records the attempts to unlock it. This tin beryllium adjuvant successful analyzing which bots are trying to entree your website. For sites that are receiving a batch of unwanted AI bot attention, this tin beryllium utilized successful discussions, sometimes legal, pinch the owners of those bots.

Cons

The cons of the server stack implementation methods are chiefly the attraction overhead. Most website servers are reasonably locked down, truthful only those who really cognize what they are doing pinch them will beryllium allowed to entree the server files, WAF aliases CDN. This intends changes to the blocks will apt request to spell done a developer, alternatively than beryllium implemented straight by an SEO. This request for an intermediary comes pinch time, resource, and costs implications, particularly if the server is managed by a 3rd statement for illustration a improvement agency.

For each furniture of security, bot spoofing is possible. Although the WAF is the strongest statement of defense, it is still imaginable that highly precocious bots tin bypass its validation checks. This intends that location is nary wholly foolproof method of blocking rogue AI bots via the server stack. However, they are still highly effective for most.

So Which Should We Use?

There is nary 1 reply to this. It is limited connected your website’s set-up, costs, and guidance structure.

In an perfect world, you would artifact the bots astatine each level of the server stack. The server is simply a bully measurement to artifact known user-agents and tin observe elemental patterns successful bot behavior. The CDN blocks are mostly effective and will forestall the bots from consuming server bandwidth. WAF is the astir effective astatine spotting spoofed bots and preventing precocious AI scrapers from accessing the site. However, you whitethorn not person easy entree to configure your WAF, if your tract has 1 astatine all.

The robots.txt is the simplest method of declaring a desire for definite bots to not entree your website, and it is effective for responsible bots. However, it tin simply beryllium ignored, and truthful is simply a deterrent, not a prevention method.

In summary, if you person a beardown request to artifact definite AI crawlers, I would urge going arsenic precocious up the server stack arsenic possible; blocking via the WAF if you can, the CDN if you can’t, and via the server arsenic a past resort.

If you only request to artifact 1 aliases 2 of the astir reputable AI crawlers, you are apt capable to conscionable trust connected the robots.txt arsenic a deterrent. However, I would besides propose monitoring your server logs to spot if immoderate of those bots are slipping past your robots.txt disallow.

More Resources: