An AI crawler reads a homepage the way a browser with JavaScript switched off does: it takes the HTML the server sends and nothing that a script draws afterwards. We read 2,000 domains' homepages that way, the top 1,000 websites and 1,000 drawn at random from ranks 10,001 to 100,000, to see how much of each page reaches the crawler, whether the page says who is behind it, and whether its llms.txt points anywhere real.
The short version
Shares are of the 441 top sites and 595 mid-sized sites that answered us with a web page. Domains with no website of their own, homepages that refused our crawler, and those whose robots.txt asked us to stay out are left out and counted; how we counted explains.
How much of the page arrives without JavaScript
94% of top homepages send their text in the HTML itself, where every crawler can read it. 0.9% send part of it and draw the rest with scripts, and 5% send an empty shell that only a browser fills in. To a crawler that runs no scripts, which is most AI crawlers today, that last group reads as a blank page.
We find no difference by size here: 5.7% of homepages in ranks 10,001 to 100,000 rely on scripts for their text, against 5.9% of the top 1,000, a difference inside the margin of error (±2.9 points).
Counting words gives the same picture from another side. The median top homepage carries 571 words of its own text, menus, header and footer left out; the median mid-sized one, 623. 14% of top homepages carry fewer than 50, whatever the reason: a search box, a sign-in screen, a page of pictures, or text drawn by scripts. To an AI crawler, each of them says almost nothing about the company.
The label comes from the HTML alone: an empty application root or a "turn on JavaScript" notice, read together with the word count. To check it, we opened 50 of these homepages, picked at random, in headless Chrome and compared the text a browser shows with the text the server sent. The two agreed on 42 of 50 (84%). On 2, the server sent fewer than 50 words of a page the browser showed in full and our label missed it, so the shares above are a floor, not a ceiling. On 5 more, the server sent real text and scripts added to it (tabs, panels, content loaded later): a crawler still gets the page, though not all of it. On 1, our label said scripts drew the text and the browser showed about the same as the server sent.
Who tells machines who they are
A page title is near universal: 99% of top homepages have one. A meta description, the one-line summary search engines and assistants often quote, is on 88%. 55% have exactly one main heading; 35% have none.
Fewer say in structured data who the company behind the page is. 45% of top homepages carry Organization markup, or one of its kinds such as Corporation, NewsMediaOrganization or LocalBusiness. That markup is how a page states its name, logo and other profiles in a form no reader has to guess at. Of the top homepages that have it in JSON-LD, 78% list their other profiles with sameAs, a median of 5 links each: the link between the website and the same company on Wikipedia, LinkedIn or a stock exchange.
| Homepages with | Top 1,000 | Ranks 10,001 to 100,000 |
|---|---|---|
| A title | 99% | 99% |
| A meta description | 88% | 82% |
| Exactly one H1 | 55% | 52% |
| Any schema.org markup | 56% | 52% |
| JSON-LD markup | 52% | 47% |
| Microdata or RDFa | 13% | 17% |
| Organization markup | 45% | 40% |
| Organization with sameAs links | 34% | 27% |
The markup types in use
56% of top homepages carry some schema.org markup, most of it as JSON-LD (52%), with microdata or RDFa on 13%. The most common type is Organization, on 37% of them.
llms.txt: who has one, and does it work
28% of top homepages publish an llms.txt file at the root of the site, and 17% of mid-sized ones. It is a short Markdown map of a site's most useful pages, written for AI agents to read. Of the 223 files we found in both samples, 73% pass the format check our audits run: a title, a one-line summary, at least one section and at least one link.
We followed up to 20 links in each file, one at a time and only where the site's robots.txt allowed us: 3,553 links in 202 files. 86% answered straight away and 12% redirected to a page that answered, which still works but costs every reader an extra request. 1.4% were dead, a 404 or 410. 1 in 11 files carried at least one dead link: a file meant to point AI at a site's best pages, pointing it at pages that no longer exist.
An llms.txt does not decide whether a site appears in AI answers; Google says its search ignores the file. Its readers are agents and tools that go looking for it. Our llms.txt guide covers what it is for and how to write one that earns its place.
The top 1,000 against the mid-sized web
The second sample is the size of company most business buyers deal with: 1,000 sites drawn at random from ranks 10,001 to 100,000. Each sample has around 518 readable homepages, so a share near 50% carries a margin of error of about ±4.3 points. A difference between the samples counts only when it is larger than its own margin.
| Homepages with | Top 1,000 | Mid-sized | Difference | Real difference? |
|---|---|---|---|---|
| Rely on scripts for text | 5.9% | 5.7% | -0.2 points | No, within ±2.9 |
| Drawn entirely by script | 5% | 5.2% | +0.2 points | No, within ±2.7 |
| A meta description | 88% | 82% | -6.4 points | Yes |
| Exactly one H1 | 55% | 52% | -2.3 points | No, within ±6.1 |
| Any schema.org markup | 56% | 52% | -3.9 points | No, within ±6.1 |
| JSON-LD markup | 52% | 47% | -5.0 points | No, within ±6.2 |
| Organization markup | 45% | 40% | -4.4 points | No, within ±6.1 |
| Organization markup with sameAs | 34% | 27% | -6.8 points | Yes |
| An llms.txt | 28% | 17% | -10.7 points | Yes |
3 of the 9 figures differ by more than chance would explain. Mid-sized homepages are less likely to have a meta description (82% against 88%), less likely to have Organization markup with sameAs (27% against 34%) and less likely to have an llms.txt (17% against 28%). For the rest, the samples are within their margins: no difference we can claim.
What to take from it
- Read your homepage the way a crawler does. Turn JavaScript off in your browser, or view the page source, and check your main message is there in words. Our guide to JavaScript sites shows how to tell, and what to fix.
- Give the page one H1 that says what you do, and a meta description that says it in a sentence.
- Add Organization markup with your name, logo and a
sameAslink to each profile that is yours. It is the plainest way to tell a machine which company is behind the site; see making your brand one clear entity and structured data for AI search. - If you publish an llms.txt, check its links after every redesign. A dead link in it sends agents to a page that is gone.
- Check your robots.txt and firewall let AI crawlers reach the homepage at all. Our robots.txt study shows how often the biggest sites keep them out.
How we counted
| The list | The Tranco list L5KW4 of 2026-11-18, the same list as our robots.txt study. Tranco is a research ranking built to resist manipulation. |
| Two samples | The top 1,000 domains, and 1,000 drawn at random from ranks 10,001 to 100,000 with the random seed 20261119, so anyone holding the list can draw the same sites again. |
| Collected | 2026-11-19, from India, by our crawler under its own name: GeoFlakesBot/1.0. Its user agent carries no address yet; it will link to the crawler's own page once our public site is live. A homepage can differ by country (a redirect to a local edition, a consent prompt in Europe): this is the view from India. |
| robots.txt first | Each site's robots.txt was read before its homepage and obeyed for our crawler by name, or for every crawler where it names none. A homepage it closed to us was not fetched (79 sites). The same rule held for llms.txt and each of its links. |
| A website means | The domain, or its www host when the domain itself had no answer, ending on a 200 response with an HTML page. Everything else is left out and counted: 347 and 237 domains with no website of their own (content delivery, DNS and tracking hosts), errors and files. |
| Refused | A 401, 403 or 429, a bot check (including a firewall's holding page with no title and almost no text), a redirect to a sign-in or a captcha, or a page that is only a cookie prompt: 117 and 136 sites. They are never counted as empty pages. |
| Counted once | Domains that end on the same address after redirects count as one website, the better-ranked one kept: 46 and 2 duplicates removed. |
| Reading the page | The fetcher and parser our audits use. Words are the page's visible text with scripts, menus, header and footer removed. The rendering label (in the HTML, partly or wholly drawn by script) reads the HTML alone, and agreed with headless Chrome on 84% of 50 homepages picked at random. |
| Markup | Types declared in JSON-LD, microdata or RDFa. Organization includes every kind of it (Corporation, LocalBusiness and its kinds, NewsMediaOrganization and others). sameAs is read from JSON-LD. |
| llms.txt | Fetched at the root of the address the homepage ended on, judged by the validator our audits run. Up to 20 links per file were followed, one request at a time. Only a 404 or 410 counts as dead; a redirect, a refusal, an error or a timeout is counted apart. |
| Margins | Each share has a 95% margin of error; with the 441 top homepages a share near 50% is good to about ±4.7 points; the share of llms.txt files with a dead link, from 202 files, to about ±4 points. A difference between samples is claimed only past its own margin. |
| Not measured | What crawlers that do run scripts see (Googlebot renders pages later), pages other than the homepage, and blocks at a firewall that answered other visitors. |
| Published | Aggregates only. No site is scored or named in the findings. |
How GEOFLAKES checks this
Every audit reads your homepage the way this study did: what the server sends without scripts, the title, description and headings, every schema.org type in every syntax, and your llms.txt against the same validator. Where the text is drawn by scripts, the report says so; where Organization markup is missing, the implementation pack writes it from facts your own pages state.
Sources
- Tranco: a research-oriented top sites ranking hardened against manipulation (Le Pochat et al., NDSS 2019)
- The rise of the AI crawler (Vercel): which AI crawlers render JavaScript
- Organization and sameAs (schema.org)
- Organization structured data (Google Search Central)
- The llms.txt proposal (llmstxt.org)
- RFC 9309: Robots Exclusion Protocol (IETF)





