GPTBot
GPTBot is a crawler run by OpenAI. It trains OpenAI foundation models.
What it does
GPTBot collects pages to train or improve AI models. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.
Model training and content-use controls. Some also govern answer grounding; review the operator's policy before changing access.
User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotThis is a full user agent string GPTBot sends. Version numbers change over time; match the GPTBot token rather than the whole string.
robots.txt
To block GPTBot, add this to the robots.txt at the root of your site:
User-agent: GPTBot
Disallow: /Blocking it asks the operator not to use your pages for model training from then on. It does not remove content already collected, and it does not stop the operator's search crawler or live fetchers, which use their own user agents.
To allow it explicitly, for example after a broader rule blocks other crawlers:
User-agent: GPTBot
Allow: /Check what your robots.txt allows for GPTBot and the other AI crawlers in this directory.
Verifying requests
OpenAI publishes the IP addresses GPTBot uses. A request that claims to be GPTBot from any other address is not genuine.
In SurfacedBy
SurfacedBy's AI traffic tracking recognizes GPTBot in the user agent and checks each request against OpenAI's published addresses. Only requests that pass count as crawls; the rest are shown separately.
Operator documentation
Details on this page come from OpenAI's own documentation: developers.openai.com/api/docs/bots.