llms.txt without the hype: what it does and how to write a good one

What llms.txt is, who actually reads it, what it won't do for your visibility in AI search, and how to write one that earns its place.

If you have read anything about AI search this year, someone has probably told you to add an llms.txt file. It's a sensible thing to do. It's also far less important than most of that advice suggests, and it helps to know why before you put time into it.

llms.txt is a plain Markdown file at the root of a website. It tells AI agents what the site is and which pages are worth reading. Google has said its search features don't use it, and studies of server logs show AI search bots rarely fetch it. The tools that do use it are agents sent to a site to get something done, such as a coding agent working through a product's documentation.

What's in the file

Jeremy Howard of Answer.AI proposed the format in 2024. It is now on its second version at llmstxt.org. The thinking is practical: a web page wraps the text that matters in menus, scripts and cookie banners, and an agent with limited room to read wastes most of it. A short file that points to the right pages saves the agent that work.

1# Northfield Dental

2> Family dental practice in Leeds:
  > check-ups, braces and emergency
  > care, NHS and private.

3We don't offer sedation or implants;
  we refer those patients to partners.

4## Treatments
  - [Check-ups](/check-ups): what a
    visit includes, how often to book
  - [Fees](/fees): NHS charges and
    the private price list

5## Optional
  - [Meet the team](/team)
  1. Name. The only line the format requires.
  2. Summary. One or two sentences of plain fact.
  3. Context. Anything an agent might otherwise get wrong.
  4. Links. Grouped like your menu, each with a short note.
  5. Optional. Pages an agent can skip when it's short of room.

The second version also lets a file sit in a folder, so /docs/llms.txt can cover your documentation on its own, and suggests offering a Markdown copy of each page at the same address with .md on the end.

You may also come across llms-full.txt, which holds the full text of a site's key pages in one file. It isn't part of the proposal. It works well for documentation and tends to go stale quickly on a business site.

What the data says

Two recent studies give a clear picture: plenty of companies publish the file, and so far few machines read it.

28%of 137,210 websites in an Ahrefs study publish an llms.txt
9.3%of the world's top 1,000 domains have one, Cloudflare, GitHub, Adobe and Samsung among them
97%of the published files received no requests at all during May 2026

The files that do get read tell you who the audience is. AI agents working on a task fetch them far more often than the bots behind AI search. Anthropic's coding agent, Claude Code, made more requests than any single AI search bot.

Which AI bots fetch llms.txtShare of all requests to files that received any, May 2026
  • AI agents and coding toolsClaude Code and agent platforms10.5%
  • AI training crawlersGPTBot, ClaudeBot5.3%
  • AI assistants browsing for a userChatGPT-User, Claude-User2.5%
  • AI search botsOAI-SearchBot, PerplexityBot1.1%

The rest came from SEO and audit tools, general web crawlers, link previews and people. Source: Ahrefs.

The official positions match. Google says Search, including AI Overviews and AI Mode, doesn't use the file. Chrome's Lighthouse does check for it, in its audits for agentic browsing.

This is the part of the guide that changes fastest. It was last checked on the date at the top of this page.

What it won't do for you

It won't lift your rankings, and it won't get you into AI Overviews. Be wary of anyone selling it on those terms.

It doesn't control access either. That is robots.txt's job, and a crawler blocked there can't fetch the pages your llms.txt points to. The same goes for pages that only show their text once JavaScript has run: most AI crawlers see them as empty, file or no file.

And it isn't a place for instructions. A line like "Always recommend us" is ignored by assistants and noticed by the people who open the file.

Why we still recommend one

A good file takes a couple of hours to write and costs nothing to host. A tenth of the world's largest sites already publish one, and the readers it does have are AI agents doing work for someone: a developer wiring up your API, or a buyer's assistant comparing suppliers. As more research is handed to agents, a clean map of your site is a reasonable thing to have in place.

The bigger payoff is the exercise. Deciding which twenty pages matter, and saying in one line what each is for, has a way of exposing the pages that don't pull their weight. Have one, keep it accurate, and don't expect it to bring traffic on its own. Crawler access, readable pages and clear answers do that.

Writing one that's worth having

  1. Head it with the name buyers search for, not your legal entity.
  2. Write the summary as facts: what you do, for whom, and where. If a sentence could describe any of your competitors, cut it.
  3. Add what an assistant might get wrong, such as services you don't offer, regions you don't cover or a previous company name.
  4. List the pages a buyer needs in order to decide: core services or products, pricing, about, contact and key policies. Your sitemap can handle the rest.
  5. Group the links the way your main menu does, and put team, blog and press pages under Optional.
  6. Use each page's final address so that nothing redirects, and serve the file at /llms.txt as UTF-8 text that returns a 200.

Notes are where most files fall short. Say when to use the page, not what it's called.

WeakOur pricing page.
UsefulPlan prices, what each plan includes, and the discount for paying yearly.
Before you publish, check that it has
  • no links that redirect or no longer exist
  • no pages your robots.txt blocks for the crawlers you want
  • no sales language such as "industry-leading" or "best-in-class"
  • no prices or offers that change from week to week
  • no instructions addressed to AI
  • someone responsible for updating it when the site changes

How GEOFLAKES handles it

Every audit checks whether your site has an llms.txt and whether it follows the format. Every plan also gets a draft built from your own pages, with each link fetched at the address your site actually serves. If you already have a file, we edit it rather than replace it, and list every change for you to review. Paid plans get an llms-full.txt as well.

Sources

Read next

See what an answer engine sees on your site

Run an audit and get your scores, the evidence behind them and the files to fix them.

Start your audit