If you have read anything about AI search this year, someone has probably told you to add an llms.txt file. It's a sensible thing to do. It's also far less important than most of that advice suggests, and it helps to know why before you put time into it.
llms.txt is a plain Markdown file at the root of a website. It tells AI agents what the site is and which pages are worth reading. Google has said its search features don't use it, and studies of server logs show AI search bots rarely fetch it. The tools that do use it are agents sent to a site to get something done, such as a coding agent working through a product's documentation.
What's in the file
Jeremy Howard of Answer.AI proposed the format in 2024. It is now on its second version at llmstxt.org. The thinking is practical: a web page wraps the text that matters in menus, scripts and cookie banners, and an agent with limited room to read wastes most of it. A short file that points to the right pages saves the agent that work.
1# Northfield Dental
2> Family dental practice in Leeds:
> check-ups, braces and emergency
> care, NHS and private.
3We don't offer sedation or implants;
we refer those patients to partners.
4## Treatments
- [Check-ups](/check-ups): what a
visit includes, how often to book
- [Fees](/fees): NHS charges and
the private price list
5## Optional
- [Meet the team](/team)
- Name. The only line the format requires.
- Summary. One or two sentences of plain fact.
- Context. Anything an agent might otherwise get wrong.
- Links. Grouped like your menu, each with a short note.
- Optional. Pages an agent can skip when it's short of room.
The second version also lets a file sit in a folder, so /docs/llms.txt can cover your documentation on its own, and suggests offering a Markdown copy of each page at the same address with .md on the end.
You may also come across llms-full.txt, which holds the full text of a site's key pages in one file. It isn't part of the proposal. It works well for documentation and tends to go stale quickly on a business site.
What the data says
Two recent studies give a clear picture: plenty of companies publish the file, and so far few machines read it.
The files that do get read tell you who the audience is. AI agents working on a task fetch them far more often than the bots behind AI search. Anthropic's coding agent, Claude Code, made more requests than any single AI search bot.
The official positions match. Google says Search, including AI Overviews and AI Mode, doesn't use the file. Chrome's Lighthouse does check for it, in its audits for agentic browsing.
This is the part of the guide that changes fastest. It was last checked on the date at the top of this page.
What it won't do for you
It won't lift your rankings, and it won't get you into AI Overviews. Be wary of anyone selling it on those terms.
It doesn't control access either. That is robots.txt's job, and a crawler blocked there can't fetch the pages your llms.txt points to. The same goes for pages that only show their text once JavaScript has run: most AI crawlers see them as empty, file or no file.
And it isn't a place for instructions. A line like "Always recommend us" is ignored by assistants and noticed by the people who open the file.
Why we still recommend one
A good file takes a couple of hours to write and costs nothing to host. A tenth of the world's largest sites already publish one, and the readers it does have are AI agents doing work for someone: a developer wiring up your API, or a buyer's assistant comparing suppliers. As more research is handed to agents, a clean map of your site is a reasonable thing to have in place.
The bigger payoff is the exercise. Deciding which twenty pages matter, and saying in one line what each is for, has a way of exposing the pages that don't pull their weight. Have one, keep it accurate, and don't expect it to bring traffic on its own. Crawler access, readable pages and clear answers do that.
Writing one that's worth having
- Head it with the name buyers search for, not your legal entity.
- Write the summary as facts: what you do, for whom, and where. If a sentence could describe any of your competitors, cut it.
- Add what an assistant might get wrong, such as services you don't offer, regions you don't cover or a previous company name.
- List the pages a buyer needs in order to decide: core services or products, pricing, about, contact and key policies. Your sitemap can handle the rest.
- Group the links the way your main menu does, and put team, blog and press pages under Optional.
- Use each page's final address so that nothing redirects, and serve the file at
/llms.txtas UTF-8 text that returns a 200.
Notes are where most files fall short. Say when to use the page, not what it's called.
- no links that redirect or no longer exist
- no pages your robots.txt blocks for the crawlers you want
- no sales language such as "industry-leading" or "best-in-class"
- no prices or offers that change from week to week
- no instructions addressed to AI
- someone responsible for updating it when the site changes
How GEOFLAKES handles it
Every audit checks whether your site has an llms.txt and whether it follows the format. Every plan also gets a draft built from your own pages, with each link fetched at the address your site actually serves. If you already have a file, we edit it rather than replace it, and list every change for you to review. Paid plans get an llms-full.txt as well.
Sources
- The /llms.txt file, v2 (llmstxt.org)
- Optimizing for generative AI features on Google Search (Google Search Central)
- The llms.txt audit (Chrome for Developers)
- We analyzed 137K sites: 97% of llms.txt files never get read (Ahrefs, June 2026)
- llms.txt adoption research report (Rankability, September 2026)