Many companies missing from AI answers aren't being ignored. They're blocking the door without knowing it. A robots.txt rule written years ago, a security setting at the CDN, or a blanket block added during the 2023 worries about AI training can each keep answer engines away from every page you publish.
Three kinds of AI bot
The large AI companies now run separate bots for separate jobs, and robots.txt lets you treat each one differently. Knowing which is which is most of the work.
Collect pages that may be used to train future models. Blocking them doesn't remove you from AI search.
GPTBot, ClaudeBotBuild the index an answer engine searches when someone asks a question. Block these and you leave that engine's answers.
OAI-SearchBot, Claude-SearchBot, PerplexityBotOpen a page because a person asked about it just now. OpenAI and Perplexity say these may not follow robots.txt.
ChatGPT-User, Claude-User, Perplexity-UserCrawler by crawler
Here is what each operator says its crawler does, in its own documentation.
| Token | What it does | If you block it |
|---|---|---|
| OAI-SearchBot | Finds pages for ChatGPT's search answers (OpenAI) | You stop appearing in ChatGPT search answers |
| GPTBot | Collects content for OpenAI's model training | Your content is marked as not for training |
| Claude-SearchBot | Indexes pages for Claude's search (Anthropic) | Less visibility in Claude's search results |
| ClaudeBot | Collects content for Anthropic's model training | Future content is left out of training |
| PerplexityBot | Indexes pages for Perplexity's answers; not used for training | You drop out of Perplexity's results |
| Googlebot | Crawls for Google Search, AI Overviews and AI Mode included | You leave Google Search altogether |
| Google-Extended | Not a crawler but a control: Gemini training and grounding in Gemini apps | No effect on Google Search or its rankings |
Block training, stay in AI search
If your legal or content team wants your pages kept out of model training, you can do it without disappearing from AI answers. A common setup looks like this:
# Answer engines: welcome
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/
One detail catches people out. A crawler with a group of its own ignores the User-agent: * group completely, so repeat any rule it should still follow, like the /admin/ line above.
Blocking training is a business decision rather than a technical one. If being recommended matters more to you than how your text is used, leave the training crawlers alone as well.
Blocks you didn't write
robots.txt isn't the only gate. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains, and many firewalls and bot management tools treat unfamiliar crawlers as threats. Your robots.txt can say welcome while the network in front of it says no.
- your live robots.txt at yoursite.com/robots.txt, not the copy in your CMS
- any
Disallow: /underUser-agent: * - AI bot or crawler settings in your CDN and firewall
- server logs showing OAI-SearchBot, Claude-SearchBot and PerplexityBot getting 200 responses
- the same checks again after every redesign or platform move
How GEOFLAKES checks this
Every audit reads your robots.txt the way each AI crawler does and marks each one allowed, blocked or partly blocked, with search and training crawlers listed apart so you can see what a block actually costs. You get your own robots.txt back with suggested changes written as comments, so nothing is reopened without your say-so. And if your server or firewall refuses our crawler, the report says so, because AI crawlers are likely meeting the same wall.
Sources
- Overview of OpenAI crawlers (OpenAI)
- Does Anthropic crawl data from the web? (Anthropic)
- Perplexity crawlers (Perplexity)
- Google's common crawlers (Google)
- Cloudflare blocks AI crawlers by default (Cloudflare, July 2025)
- RFC 9309: Robots Exclusion Protocol (IETF)
