Google Recommends Using 304 Status Code To Conserve Crawl Budget

Google updated its crawl fund archiving pinch caller accusation astir really Google’s different crawlers each stock a website’s crawl capacity and added a proposal to usage a 304 server consequence to trim the magnitude of server resources utilized by crawlers.

Both of these additions are to Google’s Optimize Your Crawl Budget Documentation that is for endeavor websites pinch complete a cardinal pages and mean sized websites of complete 10,000 web pages that quickly change. Although the archiving is not aimed astatine each website, the accusation connected that page is useful for knowing really crawling works.

Google’s Crawlers Share A Website’s Crawl Capacity

Google added the pursuing passage:

“While each crawler has a different crawl demand, the crawl capacity limit is shared crossed each crawlers. This intends that precocious request from 1 crawler tin trim the capacity disposable for others.”

This intends that if a tract gets crawled by Googlebot-Image and Googlebot, some of the crawlers are sharing the crawl fund for the website. The effect is that a precocious complaint of crawling by 1 bot tin trim the capacity disposable to different Google crawlers.

Google Recommends 304 Server Response

The different absorbing proposal that Google added was the proposal to usage 304 Not Modified server consequence code. According to the Mozilla Developer Network web documentation:

“The HTTP 304 Not Modified redirection consequence position codification indicates that location is nary request to retransmit the requested resources.”

The point astir the 304 Not Modified consequence codification is that location is nary redirection happening, moreover though technically each 3xx server consequence codes are classed arsenic redirection codes. Despite that method detail, thing is being redirected.

Google’s updated archiving says:

“Use HTTP caching: Support 304 (Not Modified) HTTP position codes. If a page hasn’t changed since Google past crawled it, returning a 304 codification tells Google to reuse the cached version, redeeming your server bandwidth and resources.”

A 304 Not Modified consequence tells Google’s crawlers that a page has not changed since the past clip it was crawled. The consequence is that the server doesn’t service the page to Googlebot and Googlebot tin spell attraction connected indexing different pages.

Read Google’s updated archiving here. The 2 changes described present are the only caller additions. There are different changes that present and location that make Google’s archiving much precise, which is documented here.