Skip to content

SurfacedBy crawler

SurfacedBy checks whether AI platforms can reach and read a website. Our crawler visits a site only when someone uses SurfacedBy to check it: a customer tracking their own domain, or a visitor using one of our free tools.

What it fetches

  • robots.txt and llms.txt, to report which AI crawlers a site allows.
  • The homepage and a few public pages, to read what the site says about itself.
  • Pages listed in the site's sitemap, to check how readable they are for AI.

It sends GET and HEAD requests to public pages only. It does not run JavaScript, submit forms or log in, and when a site asks it to slow down (HTTP 429 or a Retry-After header) it waits or stops.

IP addresses

  • 149.28.75.82/32

Every request our crawler sends comes from these addresses, so they are the complete allowlist. These are fixed addresses. They change only when our infrastructure changes, and this page and the machine-readable list at surfacedby.com/bot.json always show the current set. If your host refreshes allowlists automatically, point it at that list rather than a copy.

User agent

Mozilla/5.0 (compatible; SurfacedBy/1.0; +https://surfacedby.com/bot)

This is how our requests identify themselves. When a site refuses it, some pages are requested again from the same addresses with a standard browser user agent, so a rule that allows this user agent alone does not cover every request. Allow our IP addresses instead.

Allowing the crawler

If your host or CDN runs bot protection (a firewall, a challenge page, an AI crawler block), it may answer our crawler with a security page instead of your site. Your SurfacedBy dashboard then shows a notice naming the pages it could not read. Add the addresses above to your protection's allowlist, or send them to your host's support team and ask them to allowlist the SurfacedBy crawler. Allow the addresses rather than a user agent.

While our crawler is refused, a page's title and description may be read through a data partner's crawler instead, so we can still describe your brand. Allowlisting our addresses makes that unnecessary.

The same protection can refuse the AI platforms' own crawlers. If you want AI assistants to read and cite your pages, check that their crawlers are allowed too.

Stopping the crawler

To stop our crawler visiting your site, block the addresses above or email [email protected] with your domain.