Who lets AI in? robots.txt at the world's top 1,000 websites

We read the robots.txt of the most visited websites, crawler by crawler: who keeps AI out of training, who keeps it out of search, and who does it by accident.

Every AI company now runs separate crawlers for training its models and for answering questions. A site's robots.txt can welcome one and turn away the other. We read the robots.txt of the 1,000 most visited domains on the web, crawler by crawler, to see what the biggest sites actually decided.

The short version

23%block at least one AI training crawler
13%block at least one AI search crawler, which keeps them out of that engine's answers
32%of those search blocks come from one rule for every crawler, not a named one

Shares are of the 582 sites whose robots.txt we could read. The rest of the list is mostly infrastructure with no website of its own, such as content delivery and DNS domains; how we counted explains.

Crawler by crawler

Training crawlers are turned away more often than the crawlers behind AI search: 23% of sites block at least one, against 13% for AI search. GPTBot is blocked by 15% of sites; OAI-SearchBot, which finds pages for ChatGPT's search answers, by 8.1%.

Share of sites blocking each crawlerA full block for that crawler, named or through the rule for every crawler. Scale 0 to 20%.
Model trainingAI searchFetches for a user
  • Bytespider19%
  • CCBot17%
  • ClaudeBot16%
  • GPTBot15%
  • Meta-ExternalAgent14%
  • Applebot-Extended13%
  • Google-Extended13%
  • PerplexityBot12%
  • Perplexity-User12%
  • Claude-User11%
  • ChatGPT-User10%
  • Claude-SearchBot9.6%
  • OAI-SearchBot8.1%

Googlebot and Bingbot, the classic search crawlers, are blocked by 2.6% and 2.7% of sites.

Training and search are separate decisions now

Of the sites that block GPTBot, 45% still let OAI-SearchBot in. They keep their pages out of OpenAI's model training while staying in ChatGPT's search answers, which is the split OpenAI's crawler documentation was written to allow. The rest block both, whether they meant to or not.

Blanket rules do some of the blocking

1 in 3 of the sites that keep an AI search crawler out do it through the rule for every crawler, User-agent: *, rather than by naming it. A rule like that was usually written for scrapers or a staging site, long before AI search existed, and it now decides whether ChatGPT, Claude and Perplexity can cite the site.

Newer crawlers are often missed

Anthropic's training crawler, ClaudeBot, is named in 22% of robots.txt files. Its search crawler, Claude-SearchBot, is named in 13%. Wherever a file names one and not the other, the rule for every crawler decides the unnamed one's access by default.

The biggest sites against the rest

Among the top 100 sites we could read, 23% block both AI training and AI search crawlers; in ranks 501 to 1,000 it is 8.3%. The biggest sites are the most likely to keep AI search out entirely, often the publishers whose pages answer engines would most like to quote.

What each group of sites decidedShare of readable sites in each rank band
Allow AIBlock training onlyBlock training and AI searchBlock AI search onlyBlock Google too
Ranks 1 to 10056 sites 70%23%
Ranks 101 to 500225 sites 78%12%8%
Ranks 501 to 1000301 sites 78%9.6%8.3%

What to take from it

  1. Read your own robots.txt crawler by crawler, not line by line. Our robots.txt guide lists what each one does.
  2. If you block training, check you haven't blocked search with it. Name OAI-SearchBot, Claude-SearchBot and PerplexityBot in a group of their own.
  3. Look for a User-agent: * group that blocks everything. It decides for every crawler you haven't named, including ones that launched after it was written.
  4. Check your firewall and CDN as well. Robots.txt can't show a block that happens before the request reaches your site.

How we counted

The listThe Tranco list L5KW4 of 2026-11-18, top 1,000 domains. Tranco is a research ranking built to resist manipulation.
Collected2026-11-19, one request per site, by our crawler under its own name: GeoFlakesBot/1.0
Which hostThe bare domain first, then its www host when the bare domain gave no robots.txt file. Many sites live only at www.
ReadingThe same reader our audits use, following RFC 9309: a crawler with a group of its own ignores the * group.
Blocked meansA full block (Disallow: /) for that crawler. Partial blocks are not counted. A site with no robots.txt allows every crawler.
Left out418 domains whose robots.txt we could not read on either host: mostly infrastructure with no website, and 59 that refused our crawler outright. They are left out of every share rather than guessed.
Crawler groupsEach operator's own documentation: training GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Google-Extended, Applebot-Extended; AI search OAI-SearchBot, Claude-SearchBot, PerplexityBot.
Not measuredBlocks at a firewall or CDN, which robots.txt cannot show. Real blocking is likely higher than these figures.
PublishedAggregates only. No site is scored or named in the findings.

How GEOFLAKES checks this

Every audit reads your robots.txt the way this study did, crawler by crawler, and marks each one allowed, blocked or partly blocked, with training and search crawlers listed apart. You get your own file back with suggested changes written as comments, so nothing is reopened without your say-so.

Sources

Share the findings

Drawn from this study, ready to post or pass on.

Stat card: the split between OpenAI's training crawler and its search crawler, with the study's figure.
Blocking GPTBot doesn't block ChatGPT search Stat card Download PNG
Stat card: the share of top websites blocking AI training crawlers, against those blocking AI search.
How many top sites block AI training Stat card Download PNG
A seven-slide carousel of the robots.txt study's findings, with a five-minute check to finish.
Who lets AI in? The study in seven slides Carousel7 slides Download PDF

Read next

See what an answer engine sees on your site

Run an audit and get your scores, the evidence behind them and the files to fix them.

Start your audit