Letting AI crawlers in: robots.txt for the bots that matter

Which AI crawlers do what, which to allow if you want to be cited, and how to keep your content out of training without leaving AI search.

Many companies missing from AI answers aren't being ignored. They're blocking the door without knowing it. A robots.txt rule written years ago, a security setting at the CDN, or a blanket block added during the 2023 worries about AI training can each keep answer engines away from every page you publish.

Three kinds of AI bot

The large AI companies now run separate bots for separate jobs, and robots.txt lets you treat each one differently. Knowing which is which is most of the work.

Training crawlers

Collect pages that may be used to train future models. Blocking them doesn't remove you from AI search.

GPTBot, ClaudeBot
Search crawlers

Build the index an answer engine searches when someone asks a question. Block these and you leave that engine's answers.

OAI-SearchBot, Claude-SearchBot, PerplexityBot
User fetchers

Open a page because a person asked about it just now. OpenAI and Perplexity say these may not follow robots.txt.

ChatGPT-User, Claude-User, Perplexity-User

Crawler by crawler

Here is what each operator says its crawler does, in its own documentation.

TokenWhat it doesIf you block it
OAI-SearchBotFinds pages for ChatGPT's search answers (OpenAI)You stop appearing in ChatGPT search answers
GPTBotCollects content for OpenAI's model trainingYour content is marked as not for training
Claude-SearchBotIndexes pages for Claude's search (Anthropic)Less visibility in Claude's search results
ClaudeBotCollects content for Anthropic's model trainingFuture content is left out of training
PerplexityBotIndexes pages for Perplexity's answers; not used for trainingYou drop out of Perplexity's results
GooglebotCrawls for Google Search, AI Overviews and AI Mode includedYou leave Google Search altogether
Google-ExtendedNot a crawler but a control: Gemini training and grounding in Gemini appsNo effect on Google Search or its rankings

Block training, stay in AI search

If your legal or content team wants your pages kept out of model training, you can do it without disappearing from AI answers. A common setup looks like this:

# Answer engines: welcome
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

One detail catches people out. A crawler with a group of its own ignores the User-agent: * group completely, so repeat any rule it should still follow, like the /admin/ line above.

Blocking training is a business decision rather than a technical one. If being recommended matters more to you than how your text is used, leave the training crawlers alone as well.

Blocks you didn't write

robots.txt isn't the only gate. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains, and many firewalls and bot management tools treat unfamiliar crawlers as threats. Your robots.txt can say welcome while the network in front of it says no.

Worth checking this week
  • your live robots.txt at yoursite.com/robots.txt, not the copy in your CMS
  • any Disallow: / under User-agent: *
  • AI bot or crawler settings in your CDN and firewall
  • server logs showing OAI-SearchBot, Claude-SearchBot and PerplexityBot getting 200 responses
  • the same checks again after every redesign or platform move

How GEOFLAKES checks this

Every audit reads your robots.txt the way each AI crawler does and marks each one allowed, blocked or partly blocked, with search and training crawlers listed apart so you can see what a block actually costs. You get your own robots.txt back with suggested changes written as comments, so nothing is reopened without your say-so. And if your server or firewall refuses our crawler, the report says so, because AI crawlers are likely meeting the same wall.

Sources

Share the findings

Drawn from this guide, ready to post or pass on.

Checklist card: five things to check in your robots.txt and CDN so AI search can reach you.
The five-minute robots.txt check Checklist Download PNG

Read next

See what an answer engine sees on your site

Run an audit and get your scores, the evidence behind them and the files to fix them.

Start your audit