Skip to content

AI crawlers

The 29 crawlers and fetchers AI companies use to read websites, from 15 operators. Each page explains what the crawler does, how to allow or block it in robots.txt, and how to tell its real requests from imitations. Every entry is taken from the operator's own documentation.

Check which of them your robots.txt allows

Live citation

Page fetches requested by AI platform users. A verified request establishes a page read, not a citation or a human visit.

Search indexing

Crawlers that build search indexes used by AI platforms. Access supports discovery but does not guarantee a listing or citation.

Training

Model training and content-use controls. Some also govern answer grounding; review the operator's policy before changing access.

robots.txt controls

Tokens that never appear in a request. The operator's regular crawler reads them in your robots.txt to decide how it may use your pages.

Common Questions

What is an AI crawler?
A program an AI company uses to request pages from websites. Some collect pages to train models, some build the search indexes AI platforms answer from, and some fetch a single page when a user asks about it.
Should I block AI crawlers?
It depends on the crawler. Blocking a search indexing or live citation crawler can keep your pages out of that platform's answers. Blocking a training crawler or control token usually does not, though some operators also use them for answer grounding. Each crawler's page describes what blocking it changes.
How do I block an AI crawler?

Add a rule for its user agent token to your robots.txt, for example:

User-agent: GPTBot Disallow: /

Operators that follow robots.txt usually pick up the change within a day. Some user-requested fetchers do not follow robots.txt; their pages say so.

How can I tell if a request really comes from an AI crawler?
A user agent string can be copied by anyone. Many operators publish the IP addresses their crawlers use, and some sign their requests so the signature can be checked. Each crawler's page lists the method its operator documents.