Apple Updates Applebot Docs To Include Siri AI & AI Features

Apple has made changes to its archiving for AppleBot to see crawling and usage for its AI efforts. There were different changes made to the archive connected June 08, 2026 arsenic well. Those see the additions of nosnippet controls, crawl delay, X-Robots-Tag, accessibility and more.

The About Applebot archive added a caller conception connected AI, it reads:

Applebot crawled information whitethorn beryllium utilized to supply further discourse and up-to-date contented erstwhile AI models are utilized to make output for show successful Apple products and services. For example, answering wide world knowledge questions successful Siri and Search that whitethorn see links to sources and websites utilized to thief make the answer. Web publishers tin opt retired of their contented being utilized successful these wide world knowledge answers by applying the nosnippet meta tag to circumstantial content.

In summation to the AI updates, Apple besides added specifications astir nosnippet controls. Here are the sections that were updated related to that:

Even if you disallow Applebot-Extended and tag website contented pinch the nosnippet meta tag, your website instructions whitethorn still let Applebot to crawl your webpages. Your contented will stay discoverable done Spotlight, Siri, and Safari, arsenic good arsenic different system-wide features connected Apple devices.

Apple will not usage information tagged nosnippet arsenic further discourse and up-to-date contented erstwhile AI models are utilized to make output for show successful Apple products and services.

There is besides a conception connected crawl hold that was added, specifically that Applebot does not travel crawl-delay.

Applebot does not travel crawl-delay.

Applebot is engineered for ratio and will set to minimize the effect connected tract owners. For example, to debar overloading tract servers, Applebot’s crawl complaint adjusts automatically erstwhile a tract slows down aliases returns errors. Apple besides caches crawled contented to trim unnecessary crawling. Identifying contented that doesn't request to beryllium crawled lowers infrastructure costs for tract owners and makes the net much businesslike arsenic a whole.

There is simply a caller conception connected X-Robots-Tag:

Using the X-Robots-Tag HTTP header Applebot besides supports indexing directives delivered via the X-Robots-Tag HTTP consequence header. This is useful for non-HTML resources (such arsenic PDFs aliases images) wherever meta tags can’t beryllium used, aliases erstwhile you want to use directives astatine the server level.To use a directive to Applebot: X-Robots-Tag: applebot: nosnippet Marking paywalled content Applebot supports the schema.org isAccessibleForFree spot to place pages that are down a paywall, metered access, aliases subscription. Add the spot to your page-level system information successful JSON-LD:

And a caller conception connected Accessibility:

Pages marked isAccessibleForFree: mendacious are eligible to look successful hunt results, but Applebot will not usage that contented arsenic further discourse erstwhile AI models are utilized to make output for show successful Apple products and services. This awesome applies astatine the page level. Section-level markup utilizing hasPart is not supported. To opt retired of having your contented utilized to train Apple’s instauration models, usage Applebot-Extended described successful the adjacent section.

Those are astir of the changes to this document.

Forum chat astatine X.