Applerobots.txt controlApplebot-Extended
Applebot-Extended is a robots.txt control from Apple, not a separate crawler. It controls whether Apple may use your pages for AI training.
What it does
Apple fetches pages with Applebot and checks the Applebot-Extended group in your robots.txt to decide how it may use them. Blocking Applebot-Extended does not stop Applebot crawling your site.
Model training and content-use controls. Some also govern answer grounding; review the operator's policy before changing access.
robots.txt
To block Applebot-Extended, add this to the robots.txt at the root of your site:
User-agent: Applebot-Extended
Disallow: /Blocking it asks the operator not to use your pages for model training from then on. It does not remove content already collected, and it does not stop the operator's search crawler or live fetchers, which use their own user agents.
To allow it explicitly, for example after a broader rule blocks other crawlers:
User-agent: Applebot-Extended
Allow: /Check what your robots.txt allows for Applebot-Extended and the other AI crawlers in this directory.
In SurfacedBy
Applebot-Extended never appears in a request, so there is no traffic to count. SurfacedBy's robots.txt checks report whether your site allows it, alongside Applebot, which makes the requests.
Operator documentation
Details on this page come from Apple's own documentation: support.apple.com/en-us/119829.