Google’s John Mueller answered a inquiry concerning whether it’s fine to obstacle Googlebot from indexing an age-verification popup that website visitors are required to see. His answer offers several clarity concerning whether it’s fine to purposely conceal several kinds of satisfied from Googlebot and explains how location owners can do it.
Question About Hiding Age Verification Popup
Google requires publishers to authorize Googlebot to crawl everything that a person power see but that wasn’t continually the case. There was a period in the first days of SEO anywhere SEOs blocked Google from downloading JavaScript and CSS records in command to optimize the crawl prosperity and advance improved website indexing.
But there came a period whenever Google began rendering web pages and subsequently required location owners to authorize Googlebot to download all resources that a person would see. Some sites that continued to obstacle Google from downloading crucial resources temporarily misplaced their rankings.
So the inquiry concerning whether it’s fine to obstacle Google from indexing a popup is reasonable.
The SEO asked on Reddit whether it’s fine to use robots.txt to forestall Googlebot from indexing an age-verification popup that a website is legally required to display to visitors.
The individual described a hypothetical website that loads an age-verification popup from a distinct domain through an API. The inquiry was whether blocking Googlebot from crawling that external domain would forestall the popup from being rendered.
They asked:
“Say I have a website websiteA.com – for lawful reasons it has an age verification pop up that sits on a distinct domain and is inserted on the leaf via an API. Lets call it verificationpopup.com
Lets say I wanted to obstacle Google from seeing this – if I added a disallow on verificationpopup.com/robots.txt would that forestall Google from rendering the pop up?
(this is purley a hypothetical and apologies for the foolish Q, I am not extremely tech minded)”
Mueller Says Blocking This Specific Popup Is Fine
Mueller confirmed that using robots.txt to forestall Googlebot from rendering the age-verification popup would mostly work. He additionally suggested an substitute method that involves blocking a JavaScript document liable for loading the popup.
He responded:
“That would mostly work. If you’re using a particular JS manuscript document to do this, you could additionally disallow that particular javascript document via robots.txt. (If you lone have one JS document that does a bunch of things, I wouldnt’ do that”
Mueller’s reply indicates that Google’s algorithms may be looser than it used to be, at smallest toward item akin this age verification popup. His answer confirmed that location owners can use robots.txt to forestall Googlebot from rendering an age-verification popup, equal whenever the popup is displayed to website visitors.
He returned hours afterward to detailed in additional item concerning two ways to accomplish this, addressing two ways the popup is loaded and which resources the location owner can control.
Two Ways To Prevent Googlebot From Rendering The Popup
Mueller explained that the archetypal method involves blocking Googlebot from crawling the external domain that provides the age-verification popup.
In the circumstance of an external domain that’s the origin of the asset that needs blocking, the external domain’s robots.txt can disallow crawling and Googlebot volition not fetch those resources and hence cannot use them to render the popup. Mueller stated that this is average alongside resources connected to promotion anywhere a JavaScript document liable for showing ads is blocked by robots.txt.
Mueller explained:
“To detailed the two options:
the embedded site’s satisfied is blocked by robots.txt. If you embed satisfied from verificationpopup-com on your location (say via iframe or by calling a manuscript on verificationpopup-com, etc), and verificationpopup-com disallows all crawling, afterward crawlers that prosecute robots.txt (like Googlebot) would not fetch that content, and would not use it in rendering.
This is how many ads services work. Googlebot wouldn’t crawl these URLs akin a distinct page, but it would try to fetch them during the way of rendering, and if robots.txt blocks it, afterward it doesn’t fetch the content, and can’t use it. You need to create certain that your pages motionless burden whenever those URLs are not fetched, but for item akin this, that’s normally not a problem.
Your location uses a particular JS document to burden this, and you obstacle that yourself. For example, perchance you use a “load-verification-popup.js” document to do this. If you obstacle that leaf alongside robots.txt on your site, afterward the manuscript won’t be loaded during rendering (blocked by robots.txt).
In my experience, this is small common, but it’s an option. If the robots.txt on verificationpopup-com is not item you can influence, you’d need to do this, if you wanted to forestall it from being visible/used during rendering.”
So to recap, the second method involves blocking a JavaScript document that’s used to burden the popup that’s on another domain anywhere using a robots.txt is not an option.
Takeaway
The engaging part of Mueller’s answer is that he confirmed a way to deliberately forestall Googlebot from rendering an age-verification popup that individual visitors are required to see.
He additionally explained that the website must continue to burden correctly whenever Googlebot cannot fetch the blocked resources. For an age-verification popup akin the one described in the question, he indicated that this is normally not a problem.
The general takeaway is that it’s fine to use robots.txt to forestall Googlebot from downloading and rendering an age-verification popup, equal although individual visitors are required to see it. Mueller’s answer shows that website owners exercise several authority complete which resources Googlebot uses during rendering, including resources that are a part of the cognition presented to visitors.
Featured Image by Shutterstock/TierneyMJ
.png)
