CCBot

CCBot collects pages to train or improve AI models. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.

Model training and content-use controls. Some also govern answer grounding; review the operator's policy before changing access.

Requests from CCBot carry this token in their user agent string. Match the token rather than a full string, which changes as Common Crawl updates versions.

To block CCBot, add this to the robots.txt at the root of your site:

Blocking it asks the operator not to use your pages for model training from then on. It does not remove content already collected, and it does not stop the operator's search crawler or live fetchers, which use their own user agents.

To allow it explicitly, for example after a broader rule blocks other crawlers:

Check what your robots.txt allows for CCBot and the other AI crawlers in this directory.

Common Crawl publishes the IP addresses CCBot uses. A request that claims to be CCBot from any other address is not genuine.

• index.commoncrawl.org/ccbot.json

SurfacedBy's AI traffic tracking recognizes CCBot in the user agent and checks each request against Common Crawl's published addresses. Only requests that pass count as crawls; the rest are shown separately.

Details on this page come from Common Crawl's own documentation: commoncrawl.org/ccbot.