Skip to content
ProRataSearch indexing

ProRataInc

ProRataInc is a crawler run by ProRata. It collects licensed and public content for the Gist AI answer engine.

What it does

ProRataInc crawls pages to build the search index an AI platform answers from. Most operators use a separate user agent for each job, so you can usually allow or block one without affecting the others.

Crawlers that build search indexes used by AI platforms. Access supports discovery but does not guarantee a listing or citation.

User agent

ProRataInc

Requests from ProRataInc carry this token in their user agent string. Match the token rather than a full string, which changes as ProRata updates versions.

robots.txt

To block ProRataInc, add this to the robots.txt at the root of your site:

User-agent: ProRataInc Disallow: /

Blocking it keeps your pages out of that index, so the platform has less of your content to find and cite. Allowing it makes your pages eligible, not guaranteed, to appear in answers.

To allow it explicitly, for example after a broader rule blocks other crawlers:

User-agent: ProRataInc Allow: /

Check what your robots.txt allows for ProRataInc and the other AI crawlers in this directory.

Verifying requests

ProRata does not publish a way to verify ProRataInc's requests, so any client can send this user agent. Treat traffic with this name as a claim, not proof.

In SurfacedBy

SurfacedBy's AI traffic tracking recognizes ProRataInc in the user agent. Because SurfacedBy cannot confirm these requests come from ProRata, they are shown separately as unverified and are not counted as crawls.

Operator documentation

Details on this page come from ProRata's own documentation: platform.gist.ai/docs/prorata-ai-crawler-bot.

Common Questions

What is ProRataInc?
ProRataInc is a crawler run by ProRata. It collects licensed and public content for the Gist AI answer engine.
How do I block ProRataInc in robots.txt?
Add a group for ProRataInc with "Disallow: /" to the robots.txt at the root of your site. Blocking it keeps your pages out of that index, so the platform has less of your content to find and cite. Allowing it makes your pages eligible, not guaranteed, to appear in answers.
How can I tell if a request is really ProRataInc?
ProRata does not publish a way to verify ProRataInc's requests, so any client can send this user agent. Treat traffic with this name as a claim, not proof.