Readable to AI? What a crawler finds on 2,000 homepages

We fetched 2,000 homepages the way an AI crawler does: how much text arrives without JavaScript, who marks up who they are, and how llms.txt files hold up.

An AI crawler reads a homepage the way a browser with JavaScript switched off does: it takes the HTML the server sends and nothing that a script draws afterwards. We read 2,000 domains' homepages that way, the top 1,000 websites and 1,000 drawn at random from ranks 10,001 to 100,000, to see how much of each page reaches the crawler, whether the page says who is behind it, and whether its llms.txt points anywhere real.

The short version

5.9%of top homepages send little or none of their text without JavaScript
45%say who they are with Organization markup
28%publish an llms.txt, and 9.4% of the files we checked link to a page that is gone

Shares are of the 441 top sites and 595 mid-sized sites that answered us with a web page. Domains with no website of their own, homepages that refused our crawler, and those whose robots.txt asked us to stay out are left out and counted; how we counted explains.

How much of the page arrives without JavaScript

94% of top homepages send their text in the HTML itself, where every crawler can read it. 0.9% send part of it and draw the rest with scripts, and 5% send an empty shell that only a browser fills in. To a crawler that runs no scripts, which is most AI crawlers today, that last group reads as a blank page.

We find no difference by size here: 5.7% of homepages in ranks 10,001 to 100,000 rely on scripts for their text, against 5.9% of the top 1,000, a difference inside the margin of error (±2.9 points).

How each homepage sends its textShare of homepages, as our audits label them
In the HTMLPartly drawn by scriptDrawn by script
Top 1,000441 sites 94%
Ranks 10,001 to 100,000595 sites 94%

Counting words gives the same picture from another side. The median top homepage carries 571 words of its own text, menus, header and footer left out; the median mid-sized one, 623. 14% of top homepages carry fewer than 50, whatever the reason: a search box, a sign-in screen, a page of pictures, or text drawn by scripts. To an AI crawler, each of them says almost nothing about the company.

Words of text on the homepageVisible words the server sends, menus, header and footer left out
0 to 4950 to 199200 to 499500 to 9991,000+
Top 1,000median 571 words 14%10%21%27%28%
Ranks 10,001 to 100,000median 623 words 12%10%20%26%32%

The label comes from the HTML alone: an empty application root or a "turn on JavaScript" notice, read together with the word count. To check it, we opened 50 of these homepages, picked at random, in headless Chrome and compared the text a browser shows with the text the server sent. The two agreed on 42 of 50 (84%). On 2, the server sent fewer than 50 words of a page the browser showed in full and our label missed it, so the shares above are a floor, not a ceiling. On 5 more, the server sent real text and scripts added to it (tabs, panels, content loaded later): a crawler still gets the page, though not all of it. On 1, our label said scripts drew the text and the browser showed about the same as the server sent.

Who tells machines who they are

A page title is near universal: 99% of top homepages have one. A meta description, the one-line summary search engines and assistants often quote, is on 88%. 55% have exactly one main heading; 35% have none.

Fewer say in structured data who the company behind the page is. 45% of top homepages carry Organization markup, or one of its kinds such as Corporation, NewsMediaOrganization or LocalBusiness. That markup is how a page states its name, logo and other profiles in a form no reader has to guess at. Of the top homepages that have it in JSON-LD, 78% list their other profiles with sameAs, a median of 5 links each: the link between the website and the same company on Wikipedia, LinkedIn or a stock exchange.

Homepages withTop 1,000Ranks 10,001 to 100,000
A title99%99%
A meta description88%82%
Exactly one H155%52%
Any schema.org markup56%52%
JSON-LD markup52%47%
Microdata or RDFa13%17%
Organization markup45%40%
Organization with sameAs links34%27%

The markup types in use

56% of top homepages carry some schema.org markup, most of it as JSON-LD (52%), with microdata or RDFa on 13%. The most common type is Organization, on 37% of them.

Schema.org types on top homepagesShare of top homepages declaring each type, in any syntax. Supporting types such as ListItem and SearchAction are not shown. Scale 0 to 40%.
  • OrganizationMid-sized: 37%37%
  • WebSiteMid-sized: 36%35%
  • WebPageMid-sized: 19%25%
  • BreadcrumbListMid-sized: 12%11%
  • SoftwareApplicationMid-sized: 3.4%6.1%
  • CorporationMid-sized: 1.3%5.2%
  • NewsMediaOrganizationMid-sized: 2.5%4.5%
  • FAQPageMid-sized: 5.2%3.6%
  • ItemListMid-sized: 2.4%3.6%
  • CollectionPageMid-sized: 3.4%2.3%
  • ArticleMid-sized: 2.7%1.8%
  • VideoObjectMid-sized: 0%1.8%

llms.txt: who has one, and does it work

28% of top homepages publish an llms.txt file at the root of the site, and 17% of mid-sized ones. It is a short Markdown map of a site's most useful pages, written for AI agents to read. Of the 223 files we found in both samples, 73% pass the format check our audits run: a title, a one-line summary, at least one section and at least one link.

We followed up to 20 links in each file, one at a time and only where the site's robots.txt allowed us: 3,553 links in 202 files. 86% answered straight away and 12% redirected to a page that answered, which still works but costs every reader an extra request. 1.4% were dead, a 404 or 410. 1 in 11 files carried at least one dead link: a file meant to point AI at a site's best pages, pointing it at pages that no longer exist.

What happened to the links in llms.txt files3,553 links followed, both samples together
AnsweredRedirected within the siteRedirected to another siteDead (404 or 410)Refused usError or no answer
All links202 files 86%9.6%

An llms.txt does not decide whether a site appears in AI answers; Google says its search ignores the file. Its readers are agents and tools that go looking for it. Our llms.txt guide covers what it is for and how to write one that earns its place.

The top 1,000 against the mid-sized web

The second sample is the size of company most business buyers deal with: 1,000 sites drawn at random from ranks 10,001 to 100,000. Each sample has around 518 readable homepages, so a share near 50% carries a margin of error of about ±4.3 points. A difference between the samples counts only when it is larger than its own margin.

Homepages withTop 1,000Mid-sizedDifferenceReal difference?
Rely on scripts for text5.9%5.7%-0.2 pointsNo, within ±2.9
Drawn entirely by script5%5.2%+0.2 pointsNo, within ±2.7
A meta description88%82%-6.4 pointsYes
Exactly one H155%52%-2.3 pointsNo, within ±6.1
Any schema.org markup56%52%-3.9 pointsNo, within ±6.1
JSON-LD markup52%47%-5.0 pointsNo, within ±6.2
Organization markup45%40%-4.4 pointsNo, within ±6.1
Organization markup with sameAs34%27%-6.8 pointsYes
An llms.txt28%17%-10.7 pointsYes

3 of the 9 figures differ by more than chance would explain. Mid-sized homepages are less likely to have a meta description (82% against 88%), less likely to have Organization markup with sameAs (27% against 34%) and less likely to have an llms.txt (17% against 28%). For the rest, the samples are within their margins: no difference we can claim.

What to take from it

  1. Read your homepage the way a crawler does. Turn JavaScript off in your browser, or view the page source, and check your main message is there in words. Our guide to JavaScript sites shows how to tell, and what to fix.
  2. Give the page one H1 that says what you do, and a meta description that says it in a sentence.
  3. Add Organization markup with your name, logo and a sameAs link to each profile that is yours. It is the plainest way to tell a machine which company is behind the site; see making your brand one clear entity and structured data for AI search.
  4. If you publish an llms.txt, check its links after every redesign. A dead link in it sends agents to a page that is gone.
  5. Check your robots.txt and firewall let AI crawlers reach the homepage at all. Our robots.txt study shows how often the biggest sites keep them out.

How we counted

The listThe Tranco list L5KW4 of 2026-11-18, the same list as our robots.txt study. Tranco is a research ranking built to resist manipulation.
Two samplesThe top 1,000 domains, and 1,000 drawn at random from ranks 10,001 to 100,000 with the random seed 20261119, so anyone holding the list can draw the same sites again.
Collected2026-11-19, from India, by our crawler under its own name: GeoFlakesBot/1.0. Its user agent carries no address yet; it will link to the crawler's own page once our public site is live. A homepage can differ by country (a redirect to a local edition, a consent prompt in Europe): this is the view from India.
robots.txt firstEach site's robots.txt was read before its homepage and obeyed for our crawler by name, or for every crawler where it names none. A homepage it closed to us was not fetched (79 sites). The same rule held for llms.txt and each of its links.
A website meansThe domain, or its www host when the domain itself had no answer, ending on a 200 response with an HTML page. Everything else is left out and counted: 347 and 237 domains with no website of their own (content delivery, DNS and tracking hosts), errors and files.
RefusedA 401, 403 or 429, a bot check (including a firewall's holding page with no title and almost no text), a redirect to a sign-in or a captcha, or a page that is only a cookie prompt: 117 and 136 sites. They are never counted as empty pages.
Counted onceDomains that end on the same address after redirects count as one website, the better-ranked one kept: 46 and 2 duplicates removed.
Reading the pageThe fetcher and parser our audits use. Words are the page's visible text with scripts, menus, header and footer removed. The rendering label (in the HTML, partly or wholly drawn by script) reads the HTML alone, and agreed with headless Chrome on 84% of 50 homepages picked at random.
MarkupTypes declared in JSON-LD, microdata or RDFa. Organization includes every kind of it (Corporation, LocalBusiness and its kinds, NewsMediaOrganization and others). sameAs is read from JSON-LD.
llms.txtFetched at the root of the address the homepage ended on, judged by the validator our audits run. Up to 20 links per file were followed, one request at a time. Only a 404 or 410 counts as dead; a redirect, a refusal, an error or a timeout is counted apart.
MarginsEach share has a 95% margin of error; with the 441 top homepages a share near 50% is good to about ±4.7 points; the share of llms.txt files with a dead link, from 202 files, to about ±4 points. A difference between samples is claimed only past its own margin.
Not measuredWhat crawlers that do run scripts see (Googlebot renders pages later), pages other than the homepage, and blocks at a firewall that answered other visitors.
PublishedAggregates only. No site is scored or named in the findings.

How GEOFLAKES checks this

Every audit reads your homepage the way this study did: what the server sends without scripts, the title, description and headings, every schema.org type in every syntax, and your llms.txt against the same validator. Where the text is drawn by scripts, the report says so; where Organization markup is missing, the implementation pack writes it from facts your own pages state.

Sources

Share the findings

Drawn from this study, ready to post or pass on.

Stat card: how many llms.txt files point AI at a page that no longer exists.
Dead links in llms.txt files Stat card Download PNG
Stat card: the share of top homepages that send little or none of their text without JavaScript.
How many homepages need JavaScript for their text Stat card Download PNG
Stat card: the share of homepages with Organization markup, and how many link their other profiles.
How many homepages say who they are Stat card Download PNG
A seven-slide carousel of what a crawler finds on 2,000 homepages, with a checklist to finish.
Readable to AI? The homepage study in seven slides Carousel7 slides Download PDF
Checklist card: six things to check on your homepage so AI crawlers can read and recognise it.
The homepage AI-readiness check Checklist Download PNG
Illustration: a homepage in a browser beside the same page read without JavaScript, with the study's figure.
What you see against what a crawler sees Illustration Download PNG

Read next

See what an answer engine sees on your site

Run an audit and get your scores, the evidence behind them and the files to fix them.

Start your audit