ClaudeBot
ClaudeBot is a crawler run by Anthropic. It collects web content to train Anthropic's models.
What it does
ClaudeBot collects pages to train or improve AI models. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.
Model training and content-use controls. Some also govern answer grounding; review the operator's policy before changing access.
User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])This is a full user agent string ClaudeBot sends. Version numbers change over time; match the ClaudeBot token rather than the whole string.
robots.txt
To block ClaudeBot, add this to the robots.txt at the root of your site:
User-agent: ClaudeBot
Disallow: /Blocking it asks the operator not to use your pages for model training from then on. It does not remove content already collected, and it does not stop the operator's search crawler or live fetchers, which use their own user agents.
To allow it explicitly, for example after a broader rule blocks other crawlers:
User-agent: ClaudeBot
Allow: /Check what your robots.txt allows for ClaudeBot and the other AI crawlers in this directory.
Verifying requests
Anthropic publishes the IP addresses ClaudeBot uses. A request that claims to be ClaudeBot from any other address is not genuine.
In SurfacedBy
SurfacedBy's AI traffic tracking recognizes ClaudeBot in the user agent and checks each request against Anthropic's published addresses. Only requests that pass count as crawls; the rest are shown separately.
Operator documentation
Details on this page come from Anthropic's own documentation: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler.