Every AI company now runs separate crawlers for training its models and for answering questions. A site's robots.txt can welcome one and turn away the other. We read the robots.txt of the 1,000 most visited domains on the web, crawler by crawler, to see what the biggest sites actually decided.
The short version
Shares are of the 582 sites whose robots.txt we could read. The rest of the list is mostly infrastructure with no website of its own, such as content delivery and DNS domains; how we counted explains.
Crawler by crawler
Training crawlers are turned away more often than the crawlers behind AI search: 23% of sites block at least one, against 13% for AI search. GPTBot is blocked by 15% of sites; OAI-SearchBot, which finds pages for ChatGPT's search answers, by 8.1%.
Training and search are separate decisions now
Of the sites that block GPTBot, 45% still let OAI-SearchBot in. They keep their pages out of OpenAI's model training while staying in ChatGPT's search answers, which is the split OpenAI's crawler documentation was written to allow. The rest block both, whether they meant to or not.
Blanket rules do some of the blocking
1 in 3 of the sites that keep an AI search crawler out do it through the rule for every crawler, User-agent: *, rather than by naming it. A rule like that was usually written for scrapers or a staging site, long before AI search existed, and it now decides whether ChatGPT, Claude and Perplexity can cite the site.
Newer crawlers are often missed
Anthropic's training crawler, ClaudeBot, is named in 22% of robots.txt files. Its search crawler, Claude-SearchBot, is named in 13%. Wherever a file names one and not the other, the rule for every crawler decides the unnamed one's access by default.
The biggest sites against the rest
Among the top 100 sites we could read, 23% block both AI training and AI search crawlers; in ranks 501 to 1,000 it is 8.3%. The biggest sites are the most likely to keep AI search out entirely, often the publishers whose pages answer engines would most like to quote.
What to take from it
- Read your own robots.txt crawler by crawler, not line by line. Our robots.txt guide lists what each one does.
- If you block training, check you haven't blocked search with it. Name OAI-SearchBot, Claude-SearchBot and PerplexityBot in a group of their own.
- Look for a
User-agent: *group that blocks everything. It decides for every crawler you haven't named, including ones that launched after it was written. - Check your firewall and CDN as well. Robots.txt can't show a block that happens before the request reaches your site.
How we counted
| The list | The Tranco list L5KW4 of 2026-11-18, top 1,000 domains. Tranco is a research ranking built to resist manipulation. |
| Collected | 2026-11-19, one request per site, by our crawler under its own name: GeoFlakesBot/1.0 |
| Which host | The bare domain first, then its www host when the bare domain gave no robots.txt file. Many sites live only at www. |
| Reading | The same reader our audits use, following RFC 9309: a crawler with a group of its own ignores the * group. |
| Blocked means | A full block (Disallow: /) for that crawler. Partial blocks are not counted. A site with no robots.txt allows every crawler. |
| Left out | 418 domains whose robots.txt we could not read on either host: mostly infrastructure with no website, and 59 that refused our crawler outright. They are left out of every share rather than guessed. |
| Crawler groups | Each operator's own documentation: training GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Google-Extended, Applebot-Extended; AI search OAI-SearchBot, Claude-SearchBot, PerplexityBot. |
| Not measured | Blocks at a firewall or CDN, which robots.txt cannot show. Real blocking is likely higher than these figures. |
| Published | Aggregates only. No site is scored or named in the findings. |
How GEOFLAKES checks this
Every audit reads your robots.txt the way this study did, crawler by crawler, and marks each one allowed, blocked or partly blocked, with training and search crawlers listed apart. You get your own file back with suggested changes written as comments, so nothing is reopened without your say-so.
Sources
- Tranco: a research-oriented top sites ranking hardened against manipulation (Le Pochat et al., NDSS 2019)
- RFC 9309: Robots Exclusion Protocol (IETF)
- Overview of OpenAI crawlers (OpenAI)
- Does Anthropic crawl data from the web? (Anthropic)
- Perplexity crawlers (Perplexity)


