Claude-User
Claude-User is a fetcher run by Anthropic. It fetches a page when a user pastes a link into Claude.
What it does
Claude-User fetches a page live when someone asks the AI platform about it. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.
Page fetches requested by AI platform users. A verified request establishes a page read, not a citation or a human visit.
User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])This is a full user agent string Claude-User sends. Version numbers change over time; match the Claude-User token rather than the whole string.
robots.txt
To block Claude-User, add this to the robots.txt at the root of your site:
User-agent: Claude-User
Disallow: /Blocking it stops the assistant reading your page when a user asks for it, so the answer is built from other sources. Operators can treat user-requested fetches differently from automatic crawling; check the operator's documentation for how it applies robots.txt.
To allow it explicitly, for example after a broader rule blocks other crawlers:
User-agent: Claude-User
Allow: /Check what your robots.txt allows for Claude-User and the other AI crawlers in this directory.
Verifying requests
Anthropic publishes the IP addresses Claude-User uses. A request that claims to be Claude-User from any other address is not genuine.
In SurfacedBy
SurfacedBy's AI traffic tracking recognizes Claude-User in the user agent and checks each request against Anthropic's published addresses. Only requests that pass count as crawls; the rest are shown separately.
Operator documentation
Details on this page come from Anthropic's own documentation: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler.