Welcome to the Pulse: This week, we’re looking at how to use a 503 during concise PHP place downtime, what robots.txt controls for AI crawlers and agents, and how Google handles bracket URLs and JPEG XL images.
Here’s what matters for you and your work.
Google Says Serve A 503 When Taking Sites Down For PHP Patches
In a LinkedIn post on Oct. 21, Google Search Central reminded anyone taking servers offline for PHP patches to come back a 503 position code. The article went up the day PHP released emergency fixes for two crucial flaws.
Key facts:
The article stated the 503 “ensures already indexed URLs are preserved in Search during you work” and noted to evade broad 404 or 200 error pages, and John Mueller of Google additionally shared a reminder on Bluesky saying that the 503 does not have to be served through PHP. In their crawl-rate documentation, Google advises users to bounds serving 503 position codes to no longer than 1-2 days, as they may autumn a URL from their indicator if it is returning a 503 for multiple days.
Why This Matters
If your location returns a 503 error because of downtime caused by a PHP patch, it volition have to be triggered via an being that remains accessible (i.e., during the period your PHP is unavailable), e.g., a CDN routing regulation or a impermanent server your DNS points to. According to Google’s guide to pausing a site, if you trigger a 503 in reply to robots.txt requests, all crawling is blocked.
Read our complete coverage: Google Reminds Sites To Serve A 503 During PHP Patch Downtime
Illyes Says Standard Robots.txt Rules Cover Crawling, Not Later Use
Gary Illyes of Google stated in an Oct. 21 Search Central video that according to RFC 9309, the norm robots.txt directives (robots.txt rules) lone dictate what compliant crawlers are allowed to crawl. These directives do not authority how systems use satisfied following it’s crawled.
Key facts:
Illyes called robots.txt “an admission penchant protocol” and stated that keeping one AI crawler out during allowing others method targeting that crawler’s user-agent token. Google’s list of average crawlers documents Google-Extended as the token for managing whether satisfied Google crawls may be used for training forthcoming Gemini models and for grounding in Gemini Apps and on Vertex AI, and says the token doesn’t power inclusion in Google Search. Google-Extended isn’t a distinct crawler and doesn’t have its own HTTP user-agent string. A draft that Illyes co-wrote for the Internet Engineering Task Force would add a Content-Usage regulation to robots.txt for stating use preferences, but it’s motionless a operating collection document and hasn’t been approved.
Why This Matters
A Disallow regulation under a crawler’s token can assistance keep that crawler distant from the listed paths. To bounds how Google uses crawled satisfied for Gemini, you can use Google-Extended. A norm way to province a training penchant separately from crawl admission is accessible lone as a draft.
Read our complete coverage: Google’s Illyes: Robots.txt Covers Crawling, Not AI Training
Cloudflare And OpenAI Say A Pilot Found New Pages For ChatGPT Search
Cloudflare and OpenAI say a investigation pilot helped OpenAI detect and indicator hundreds of millions of new web pages, giving ChatGPT equal additional data to diagram on.
Key facts:
The aviator drew on a participating collection of about 1% of Cloudflare’s network through signals which assistance the OpenAI crawler in identifying pages; bypassing non-existent pages and indicating if a leaf is updated. The Oct. 15 blog article (co-authored by two OpenAI employees) states that finding new pages “can initially addition crawling,” and does not study how frequently those newly identified pages are being cited, nor does it provision data concerning the figure of visits generated from the pages. Additionally, the article stated, “A indication concerning a leaf doesn’t aid approval to admission it.”
Why This Matters
OpenAI’s crawler documentation is divided by hunt vs. training. A website may use robots.txt entries akin Allow for OAI-SearchBot and Disallow for GPTBot to inform the crawler that its satisfied shouldn’t be used for training purposes. Because OpenAI ranks pages on its own, the article says, fair since your website was established through the aviator does not average you volition get a citation.
Read our complete coverage: ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot
Raw And Encoded Brackets Are Different URLs, Mueller Says
Square brackets in a URL and their percent-encoded forms, %5B and %5D, create distinct URLs, Google’s John Mueller said on Bluesky on Oct. 19. He added that he would have assumed they were equal and “probably gave group the incorrect answer on this in the past.”
Key facts:
Mueller referenced RFC 3986 (the URI syntax standard, which reserves brackets and affirmative signs) as fine as noting that if a location returned the identical satisfied in reply to either type of the URL format Mueller stated that Google would crawl all and eliminate duplicates afterward choose one. Mueller offered a solution, stating “pick one,” and recommended using it consistently throughout all pages, including links, canonical tags, sitemap entries, and organized data. This is accordant alongside the existing Google documentation on URL structure which states that reserved characters must be percent encoded.
Why This Matters
When links and canonical tags mix raw and encoded forms, Google may have to decide which type to keep. If your existing location uses raw brackets throughout, Mueller stated he’d anticipate it to keep working, although he “wouldn’t set up a new location akin this.”
Read our complete coverage: Google Crawls Raw And Encoded Bracket URLs Separately
JPEG XL Isn’t Supported In Google Images Or Discover Previews
JPEG XL images aren’t currently supported in Google Images or as previews in Discover, as Google’s John Mueller shared on Bluesky on Oct. 15.
Key facts:
His article followed JPEG XL assistance in Chrome 155, which Google began rolling out extensively on Oct. 6, and in Firefox 158 on Oct. 13. Google’s image SEO documentation lists BMP, GIF, JPEG, PNG, WebP, SVG, and AVIF as supported formats for images in an img element’s src attribute, and didn’t contain JPEG XL as of Oct. 22. Asked concerning fallbacks in a image element, Mueller stated the component includes an img element, “so that would activity for any hunt engine.”
Why This Matters
Serving JPEG XL as a image source, alongside a JPEG, WebP, or AVIF document in the img element, keeps a supported format in the src attribute, anywhere Google’s records says Search says images. Google’s Discover guide points to og:image and schema.org markup for preview images, so those should item to among the listed formats too.
Read our complete coverage: Google Images & Discover Previews Don’t Support JPEG XL
Pichai Expects AI Agents To Reach Sites By Browsing And Through Protocols
In an interview alongside Alex Heath on the Sources podcast, Google CEO Sundar Pichai shared that he expects AI agents to admission websites the two by browsing pages akin group do and through protocols designed specifically for agents.
Key facts:
“I think you have the two ways and they’re going to develop,” Pichai said. He pointed to momentum rearward delegate and fee protocols, including the Universal Commerce Protocol that Google backs. He called protocols “a far additional scalable approach” but stated he expects them to obtain longer to develop. Later in the interview, he stated he expects agents to provision a unifying cognition throughout Search and Gemini.
Why This Matters
By Pichai’s account, agents that browse pages the way group do remain in the image during protocols develop. Google’s own hosted agents browse as Google-Agent, among the user-triggered fetchers that Google says mostly disregard robots.txt rules.
Read our complete coverage: Pichai Says AI Agents Will Click Through Pages And Use Protocols and Google CEO Sundar Pichai Views Search Absorbed Into Agentic Layer
Theme Of The Week: What Each Signal Reaches
Each of the stories this week concerns a indication that crawlers and agents obtain concerning a website and anywhere they are allowed to reach. A 503 signals to Google that the outage is lone impermanent (within the recommended period example for outages from Google’s documentation). A disallowed crawl directive keeps compliant crawlers from crawling through the satisfied of your pages, but how it’s used afterward depends on controls akin Google-Extended and GPTBot. However, a Cloudflare finding indication does not provision OpenAI approval to admission your web page. Whether Google views your leaf at one URL or two depends upon whether you mix raw and encoded brackets. Currently, a JPEG XL depiction document contained in the img component cannot be viewed by Google Images. Agents that fetch according to person input autumn into an area partially exterior of robots.txt as defined by Google concerning its user-triggered fetchers. Server logs track what crawler has been making requests to your server; however, the particular crawlers and which controls govern crawling and afterward use are explained in all corresponding company’s documentation.
Top Stories Of The Week:
- Google Reminds Sites To Serve A 503 During PHP Patch Downtime
- Google’s Illyes: Robots.txt Covers Crawling, Not AI Training
- ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot
- Google Crawls Raw And Encoded Bracket URLs Separately
- Google Images & Discover Previews Don’t Support JPEG XL
- Pichai Says AI Agents Will Click Through Pages And Use Protocols
- Google CEO Sundar Pichai Views Search Absorbed Into Agentic Layer
More Resources:
- Google Launches A Video Series For Odd Technical SEO Questions
- Should I Block AI Crawlers At Robots.txt Or Server Level? – Ask An SEO
- The Expert SEO Guide To URL Parameter Handling
Featured Image: Cast Of Thousands/Shutterstock
