meta-webindexer

meta-webindexer crawls pages to build the search index an AI platform answers from. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.

Crawlers that build search indexes used by AI platforms. Access supports discovery but does not guarantee a listing or citation.

This is a full user agent string meta-webindexer sends. Version numbers change over time; match the meta-webindexer token rather than the whole string.

To block meta-webindexer, add this to the robots.txt at the root of your site:

Blocking it keeps your pages out of that index, so the platform has less of your content to find and cite. Allowing it makes your pages eligible, not guaranteed, to appear in answers.

To allow it explicitly, for example after a broader rule blocks other crawlers:

Check what your robots.txt allows for meta-webindexer and the other AI crawlers in this directory.

Meta does not publish a way to verify meta-webindexer's requests, so any client can send this user agent. Treat traffic with this name as a claim, not proof.

SurfacedBy's AI traffic tracking recognizes meta-webindexer in the user agent. Because SurfacedBy cannot confirm these requests come from Meta, they are shown separately as unverified and are not counted as crawls.

Details on this page come from Meta's own documentation: developers.facebook.com/documentation/sharing/webmasters/web-crawlers.

• Meta-ExternalAgentTraining
• Meta-ExternalFetcherLive citation