Google Says Why It May Ignore Robots.txt And Negatively Impact SEO

Google’s John Mueller answered a mobility astir robots.txt and explained an easy-to-miss correction that tin effect your SEO and website indexing goals. The circumstantial rumor was related to hunt container spam getting indexed by Google, but this correction tin hap to anyone successful wide nether immoderate context.

Website Search Box Spam

The personification who asked the mobility connected Reddit was suffering from a hunt barroom spam attack. What spammers do is hunt pinch a query that reflects their spammy niche, and they adhd a nexus aliases a website name. What happens adjacent is that the hunt barroom generates a URL that tin beryllium referenced to make the spammy hunt result.

And that’s what was happening to the personification who was asking the question. Their consequence was to adhd a statement successful the robots.txt record to forestall Google from indexing the file. But Google was indexing those spammy search-generated URLs anyway.

Google Indexed Pages Blocked By Robots.txt

Someone posted connected Reddit that their client’s Shopify hunt container was generating spammy web pages successful consequence to spammer queries and that Google was indexing them contempt a robots.txt record prohibiting Google from indexing those pages. What the customer did was redirect those spammy URLs to different web page. The personification asking the mobility didn’t inquire really to extremity the pages from being indexed (which is what they should person been asking); they asked if those redirected URLs should beryllium marked 404 instead.

The personification asked:

“Working connected a client’s Shopify shop wherever we person the /search added arsenic a disallow successful robots.txt, nevertheless these hunt results are still indexed wrong of Google.

However, if I effort to unfastened 1 of these pages, they person a redirect group and they redirect to different postulation page connected the store. Should we show a 404 page instead? What is the easiest measurement to hole this?”

Why Robots.txt File Caused Spam To Be Indexed

Google’s John Mueller took the other measurement to place and reappraisal the client’s robots.txt record and identified an correction that was causing Google to disregard the directive prohibiting Googlebot from indexing hunt results pages.

Mueller responded:

“Also, not judge if it’s your site, but the 1 I recovered pinch akin indexed URLs had sections for “user-agent: Googlebot” (in the “START: Custom Rules” artifact successful comments) arsenic good arsenic a batch much successful the “user-agent: *” conception further down. With robots.txt, the much circumstantial rules win, truthful if you person a user-agent: Googlebot section, it will *only* usage that section. If you want to use each the rules successful the “user-agent: *” section, you request to transcript them. Also, if that’s your site, past you tin conscionable database each the user-agents that you want to person shared rules for together, eg:

user-agent: googlebot

user-agent: otherbot

user-agent: imgsrc

user-agent: somethingpt

disallow: /fishes

disallow: /orange-cats

… etc …”

User-Agent Specific Directives Take Precedence

What happened is that the customer was relying connected Google to travel the directives successful a statement that’s aimed astatine each personification agents, “user-agent: *”, but because there’s different conception of the robots.txt record that’s circumstantial to Googlebot, Google ignored the “user-agent: *” directives and obeyed the 1 that specifically addressed Googlebot.

That whitethorn sound for illustration a quirk successful the measurement robots.txt works, but it really makes consciousness because this enables users to target circumstantial crawlers pinch unsocial rules and target everyone other pinch a different group of rules.

How To Be Safe From Search Box Spam

WordPress and Shopify some person ways to mitigate hunt container spam.

WordPress and Shopify some person ways to mitigate hunt container spam.

Shopify Search Box Spam Mitigation

Shopify’s website has a tutorial connected really to automatically adhd a noindex directive to each hunt results. This will efficaciously forestall indexing of each hunt results pages. However, it’s basal to not artifact hunt pages pinch robots.txt for this to work.

Using a noindex directive is much effective than utilizing robots.txt because robots.txt does not power indexing; it only controls crawling.

Shopify instructs:

“You tin hide pages that aren’t included successful your robots.txt.liquid record by customizing the <head> conception of your store’s theme.liquid layout file. You request to see immoderate metatag codification to extremity the indexing of peculiar pages.

From your Shopify admin, spell to Online Store > Themes.

Find the taxable you want to edit, click the … fastener to unfastened the actions menu, and past click Edit code.

In the layout folder, click the theme.liquid file.

To exclude the hunt template, paste the pursuing codification connected a blank statement successful the <head> section:

{% if template contains ‘search’ %}
<meta name=”robots” content=”noindex”>
{% endif %}”

WordPress Search Box Spam Mitigation

It’s rather easy to mitigate hunt container spam pinch WordPress. Users of the Yoast, Rank Math, and AIOSEO SEO plugins person their hunt results pages automatically group to noindex by default. Additionally, immoderate page builders and themes for illustration Divi (and its Extra theme) will automatically not make spammy words and URLs successful consequence to searches and alternatively will inject a fewer sentences saying that the hunt produced nary results.

Robots.txt Knowledge

Effective SEO requires a wide scope of knowledge. Robots.txt contains immoderate quirks that tin origin it to beryllium little effective than intended, truthful it’s useful to publication up connected the charismatic specifications successful bid to support up to date.

Featured Image by Shutterstock/Stockinq