SurfacedBy crawler

• Websites our customers track: robots.txt, llms.txt, sitemaps and public pages, when the site is added and again on a regular schedule to keep their reports current.
• Pages that AI answers cite, and the websites of brands those answers mention: the page itself and a few public pages such as the homepage, to see what the page says and who publishes it.
• Publisher websites: public about and contact pages, when a customer looks for who publishes a page that AI answers cite.
• Any website someone checks with our free tools: robots.txt, llms.txt, the homepage and the page they entered.

It sends GET and HEAD requests to public pages only. It does not run JavaScript, submit forms or log in, and when a site asks it to slow down (HTTP 429 or a Retry-After header) it waits or stops.

Yes. It reads robots.txt first, follows the rules for SurfacedBy (or for all crawlers when none name us), and waits between requests as long as a Crawl-delay asks. A page robots.txt disallows is not requested at all, under any user agent. When robots.txt cannot be read because the server errors or times out, it reads no pages from that site until it can (it may use a copy read in the last week). The one exception is a site whose owner has connected it to SurfacedBy, for example by installing our tracker or connecting Search Console: it is read whatever its robots.txt says, because its owner asked us to monitor it.

If your site is not connected to SurfacedBy, this robots.txt rule blocks our crawler:

Every request our crawler sends comes from these addresses, so they are the complete allowlist. These are fixed addresses. They change only when our infrastructure changes, and this page and the machine-readable list at surfacedby.com/bot.json always show the current set. If your host refreshes allowlists automatically, point it at that list rather than a copy.

robots.txt, sitemaps and pages are all requested with this user agent, from the addresses above.

Every request is signed using the Web Bot Auth standard (HTTP message signatures), so bot protection that checks signatures can confirm it comes from us. Our public keys are published at surfacedby.com/.well-known/http-message-signatures-directory.

If your host or CDN runs bot protection (a firewall, a challenge page, an AI crawler block), it may answer our crawler with a security page instead of your site. Your SurfacedBy dashboard then shows a notice naming the pages it could not read. Add the addresses above to your protection's allowlist, or send them to your host's support team and ask them to allowlist the SurfacedBy crawler. A rule that checks both the addresses and the user agent is the most reliable.

The same protection can refuse the AI platforms' own crawlers. If you want AI assistants to read and cite your pages, check that their crawlers are allowed too. The AI crawlers directory lists each one, with its user agent and how to allow it.

To stop our crawler visiting your site, add the robots.txt rule above, block the addresses above, or email [email protected] with your domain.