On July 1, Cloudflare gave every site, free tier included, a finer way to handle AI bots. Instead of one “block all AI” switch, there are now three controls, one per job a bot does: Search, Agent, and Training. Cloudflare sits in front of more than 20 percent of web domains, so most sites will inherit this without touching a setting. Two of those three purposes are the bots that fetch your pages so ChatGPT, Perplexity, and Google’s AI answers can cite you. Block them by reflex and you disappear from AI search while thinking you only stopped model training.
Read the three before you block anything
Search indexes your pages so an engine can answer about you later. Agent is the live fetch that pulls your page in real time for someone asking right now; Cloudflare files ChatGPT-User and browser agents like Gemini and Claude here. Training is the only one that is a pure rights call, because it feeds a model rather than a citation. Blocking Search or Agent is an AI-visibility decision, and it is the one most owners make without meaning to.
The trap on September 15
For new domains on ad-supported pages, Cloudflare will block Agent and Training by default and keep Search on. But a training block has a side effect: Cloudflare also blocks the multi-purpose crawlers that do Search and Training together, which it names as Googlebot, Applebot, and Bingbot. That can quietly take your Google and Bing indexing with it. Existing sites keep the choice, but the default is coming.
What to do
- Keep Search and Agent allowed unless you have a specific reason not to.
- Block Training only as a deliberate rights call, and check what else that crawler does first.
- Read your own robots.txt and WAF rules. A rule you inherited is still yours, and Cloudflare’s managed file now writes
search=yes,ai-train=no,use=reference. - Verify from the outside. “Allowed” on a settings page is a claim; a live fetch that returns your page is proof.
To see which AI bots your site allows or blocks by name, run our free robots.txt and AI bot checker. For every crawler and the job it does, here is the full crawler reference.
