Is your site blocking AI bots?
Check if your robots.txt file is preventing ChatGPT, Claude, and Gemini from reading your content.
ChatGPT-User
Google-Extended
ClaudeBot
PerplexityBotUnderstanding AI Access
What you need to know about robots.txt and large language models.
How robots.txt works
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. AI companies use specific user-agents (like GPTBot or ClaudeBot) that you can allow or disallow in this file.
Crawlers vs Retrieval
Blocking a training crawler (like GPTBot) stops OpenAI from using your data to train future models. However, user-initiated agents (like ChatGPT-User) fetch live data when a human asks a specific question.
What blocking means
If you block user-initiated AI agents, ChatGPT or Perplexity won't be able to read your live pricing pages, documentation, or recent blog posts when users ask about your product. You become invisible to AI search.
See what AI engines say about your brand
Bot access is step one. Monitor brand mentions across ChatGPT, Claude, Perplexity, and Google AI Mode, track visibility score and market rank over time, watch competitor coverage, follow AI-referred traffic, and get weekly alerts when things change.
Start 7 day trialLive queries across ChatGPT, Claude, Perplexity, and Google AI Mode.