I blocked one page in robots.txt, asked eight AI assistants to go read it, and watched the server logs. Three said no. Three went and read it anyway, and two didn’t even try.
The ones that read it weren’t breaking any rules, which took me a while to get my head around. Google has it in writing that a user’s request cancels robots.txt. Anthropic and OpenAI looked at the same file the same afternoon and decided it didn’t.
The setup
On a test server I run, each assistant got two pages of its own. One sits in a directory I disallowed for every AI user-agent I could find a name for, plus a catch-all: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Manus-User, Googlebot, Bingbot, the training crawlers, the lot. The other sits somewhere robots.txt allows. Both print a random string that exists nowhere else on the internet, so the only way to tell me that string is to have actually loaded the page.
Giving each assistant its own URL matters more than it sounds like it should. Nobody else knew those addresses existed, so anything showing up in the log belongs to exactly one prompt. Three rounds across three days, fresh URLs each time, new chat every run.
The allowed page is doing the real work. If an assistant fetches nothing, that tells you nothing, because plenty of them just don’t fetch. You only learn something when it takes the allowed page and skips the blocked one in the same sitting. I also waited until I could see assistants pulling the new robots.txt before running anything, so nobody could be blamed for a rule they’d never read.
What happened
| Assistant | Blocked page | Allowed page |
|---|---|---|
| ChatGPT | refused | fetched |
| Claude | refused | fetched |
| Meta AI | refused | fetched |
| Gemini | read it | fetched |
| Grok | read it | fetched |
| Manus | read it | fetched |
| Copilot | never arrived | never arrived |
| Perplexity | never arrived | never arrived |
Every row there comes from the server log, not from what the assistant told me it did. Those two things came apart at least once, which I’ll get to.
The ones that said no
Claude was the clearest about it. It came back with an actual error code, ROBOTS_DISALLOWED, and the message “Site disallows automated access.” In a later round it spelled it out: the fetch was blocked before the page ever loaded, and it doesn’t work around that. The log agrees. Claude-User pulled robots.txt, took the allowed page, and never went near the other one.
ChatGPT refused too, saying the URL was blocked from its fetcher. Its allowed fetch landed a minute later from Microsoft address space, so the host was clearly reachable and it just chose not to go. Both companies document the behavior, in Anthropic’s crawler docs and OpenAI’s bots page.
Meta stayed out as well, though it made a mess of explaining why. Its browsing tool told me the page didn’t exist, which isn’t true, it returns a 200 to anyone else who asks. The behavior was still right: meta-webindexer took the allowed page and never requested the blocked one. That’s a better result than it sounds, because Meta’s own docs reserve the right to ignore robots.txt when a user follows a link. It had an excuse sitting there and didn’t use it.
Gemini read it, and Google says that’s allowed
Gemini read the blocked page in every round. It shows up in the log as user-agent Google, from Google’s own address space, so there’s no guessing about who it was.
Google isn’t hiding this either. Their page on user-triggered fetchers says it flatly: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.” A person asked for it, so it counts as that person’s fetch rather than a crawl.
That’s a defensible line to take. It’s also the exact opposite of what Anthropic and OpenAI did with the same file, on the same page, within the same hour. So there’s no standard here at all, just company policy, and the policies disagree with each other.
Grok never asked in the first place
Grok read the blocked page all three rounds. It never once requested robots.txt, so there was nothing for it to obey. xAI doesn’t publish crawler docs or a user-agent token you could write a rule against, so that file is irrelevant to Grok in both directions.
Where the requests come from surprised me. Grok reaches the site two different ways. Sometimes it’s a single Google Cloud address calling itself headless Chrome. Other times a burst arrives from everywhere at once: in one round, 22 separate addresses spread over at least ten networks and six countries. Uruguay, the Netherlands, Mexico, Canada, and a dynamic residential range belonging to an ISP in Belarus. Every one of them claiming to be a Mac running Safari or Chrome.
None of those addresses is registered to xAI, and a commercial proxy service is the obvious guess. For whoever owns the site it lands the same either way. A request from a home connection in Belarus with a Safari user-agent isn’t something robots.txt can reach, and it isn’t something you’ll ever pick out of your logs as AI traffic.
The scattered pool showed up on two of the three days. On the middle day it vanished completely and every request came from that one cloud address. I don’t know what decides.
Manus read the blocked page too, arriving as plain Chrome with nothing on it to say what it was. When I ran a similar probe back in April it announced itself in the user-agent and was the easiest thing in the whole set to spot in a log. Not anymore.
Two that never showed up
Copilot and Perplexity never hit the server at all, either page, any round. Neither result says anything about robots.txt, and they failed in different ways.
Copilot can fetch things. I handed it example.com and it read back the real text. It just never got to my host, and I can’t see why from my side.
Perplexity was stranger. Given the same example.com, it described a page about vehicle MOT testing, nine numbered points, tyres, brakes, when your test is due. The actual example.com is four lines of placeholder text that IANA has kept there for years. Nothing was fetched and nothing was checked. That’s one observation rather than a rate, but it’s an assistant describing a page it never opened, on probably the least ambiguous URL there is.
What this doesn’t tell you
One domain, three rounds, one way of wording the request. A site with real history and inbound links might get treated differently, and asking through a different product surface might change things too. I only tested fetches that happen because a user asked for them, so none of this says anything about how the same companies’ training and indexing crawlers behave. Those are separate bots with separate names in that same file.
I also can’t see a refusal that happens before a request. When an assistant said no, all I know is that nothing reached my server. Whether it checked a cached robots.txt, an internal blocklist, or something else entirely, there’s no way to tell from where I’m sitting.
What to do with this if you run a site
robots.txt still works on the companies that decided it applies to them. Disallow the retrieval agents and Anthropic, OpenAI and Meta will respect it, and Claude will tell your user straight out that it was blocked. That’s real control, so set that file on purpose instead of living with whatever your CMS shipped with.
Past those three it’s closer to a polite request. Google has written itself an exemption for anything a person asks for, and no line in that file reaches an assistant that never requests it or comes in from a residential IP dressed as Safari. If robots.txt is your whole plan for controlling AI access to your content, it missed three of the eight I tested, Gemini included, and your logs won’t tell you which three.
Most people haven’t touched that file in years and it rarely says what they think it says. Our robots.txt checker reads your live one and tells you which retrieval, search and training agents you’re allowing right now. No account needed.
