Blocking the wrong AI crawler can remove you from ChatGPT and Claude search answers. Here is what each bot does, a safe robots.txt, and how to see which AI agents actually visit your site.
GPTBot is OpenAI's crawler for collecting content that may be used to train its models; ClaudeBot is Anthropic's equivalent. Neither is the bot that puts you in search answers. For that, ChatGPT uses OAI-SearchBot and Claude uses Claude-SearchBot. User-triggered fetches come from ChatGPT-User and Claude-User.
That split matters: you can block training crawlers and still stay visible in AI search, but blocking the search bots can remove you from answers.
| User agent | Company | Purpose | Block it and… |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train models | Your content should not be used in training |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search | You can drop out of ChatGPT search answers |
| ChatGPT-User | OpenAI | Visits pages when a user's action asks for it | robots.txt rules may not apply (user-initiated) |
| ClaudeBot | Anthropic | Collects content that may contribute to training | Future content excluded from training |
| Claude-SearchBot | Anthropic | Improves search result quality for Claude users | Less visibility and accuracy in Claude search |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | Claude cannot retrieve your page for users |
| PerplexityBot | Perplexity | Indexes sites for Perplexity answers | Less visibility in Perplexity |
| Google-Extended | Controls use in Gemini models (not a separate crawler) | Does not remove you from AI Overviews |
Sources: OpenAI's crawler documentation and Anthropic's crawler documentation. AI Overviews use normal Googlebot crawling; see Google's AI features guide.
Also check your CDN. Some Cloudflare and firewall settings now block AI bots by default, regardless of what robots.txt says.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlRemove the two training blocks if you are happy for your public content to inform future models; many brands allow them to improve how models describe them.
Agent analytics tools read your server or CDN logs and label AI bot visits. Thomas from OtterlyAI explains why that data is useful next to citation tracking:
Not every agent is the same. We have different on-demand AI fetchers. There are agents crawling the internet basically to build up the search index. And we also have agents from the different models that are basically crawling content for the training of the different LLM models.
An agent is crawling my content, is coming to my home page in this case, is crawling that particular page and then is deciding if it’s citation worthy content.
His practical point: compare crawl data with citation data. A page crawled often but rarely cited may need clearer, citable passages. A page cited often from few crawls is working and worth expanding.
User-agent: * disallow.Dooza is an AI-native company that builds AI products and services for small businesses. For AI search, Ranky does the work. Ranky tracks how ChatGPT, Perplexity, Gemini, Claude, Microsoft Copilot, and Google AI Overviews answer the prompts your buyers ask, maps which sources those answers cite, and then does the fixes: it drafts and publishes answer-ready pages, fixes schema, and flags the third-party threads worth joining, all behind your approval. Every Dooza product starts with a refundable pilot — 100% refund within 14 days.
Every Dooza setup starts with a crawler check: robots.txt rules per AI bot, CDN blocking, raw-HTML rendering of key pages, sitemaps, and an llms.txt file. Fixes ship before any content work, because a page bots cannot read cannot be cited. Next, see how AI citations work and LLM SEO.
Book a free pilot call to scope a refundable pilot: we check which AI bots can reach your site and fix what is blocked.
ClaudeBot is Anthropic's web crawler that collects content that may contribute to training Claude models. Blocking it signals your content should be excluded from training; it is separate from Claude-SearchBot, which supports Claude's search results.
GPTBot is OpenAI's crawler for content that may be used to train its generative AI models. Disallowing it opts your content out of training but does not remove you from ChatGPT search, which uses OAI-SearchBot.
Only if you do not want your content used for model training. Blocking GPTBot does not affect ChatGPT search visibility as long as OAI-SearchBot is allowed.
Allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, and Bingbot. These power AI search answers and user-requested page fetches.
No. Google-Extended controls use of content for Gemini models. AI Overviews rely on normal Googlebot crawling and indexing.
Filter server or CDN logs by AI user agents, verify IP ranges, use hosting dashboards like Cloudflare or Vercel, or use an agent analytics tool, and compare crawled pages with cited pages.
Automate your business with AI employees that work 24/7. Start with a refundable pilot: 100% refund within 14 days.
AI visibility tools show how ChatGPT, Perplexity, Gemini, and Google AI Overviews talk about your brand. This guide explains the metrics, how to run a free manual audit first, and how to pick a platform that turns the numbers into fixes.
Answer engine optimization is how you get mentioned and cited when people ask ChatGPT, Perplexity, Gemini, and Google AI a question. Here is how it works, what to do first, and how to measure it.
Start with a refundable pilot — 100% refund within 14 days. A Dooza engineer scopes it with you on a free 30-minute call. Pricing depends on the product; see pricing.
Refundable pilot · 100% refund within 14 days · No contracts