Claude-SearchBot
Claude-SearchBot is a crawler run by Anthropic. It indexes the web for Claude search results.
What it does
Claude-SearchBot crawls pages to build the search index an AI platform answers from. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.
Crawlers that build search indexes used by AI platforms. Access supports discovery but does not guarantee a listing or citation.
User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])This is a full user agent string Claude-SearchBot sends. Version numbers change over time; match the Claude-SearchBot token rather than the whole string.
robots.txt
To block Claude-SearchBot, add this to the robots.txt at the root of your site:
User-agent: Claude-SearchBot
Disallow: /Blocking it keeps your pages out of that index, so the platform has less of your content to find and cite. Allowing it makes your pages eligible, not guaranteed, to appear in answers.
To allow it explicitly, for example after a broader rule blocks other crawlers:
User-agent: Claude-SearchBot
Allow: /Check what your robots.txt allows for Claude-SearchBot and the other AI crawlers in this directory.
Verifying requests
Anthropic publishes the IP addresses Claude-SearchBot uses. A request that claims to be Claude-SearchBot from any other address is not genuine.
In SurfacedBy
SurfacedBy's AI traffic tracking recognizes Claude-SearchBot in the user agent and checks each request against Anthropic's published addresses. Only requests that pass count as crawls; the rest are shown separately.
Operator documentation
Details on this page come from Anthropic's own documentation: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler.