llms.txt is a short Markdown file at the root of a website (yoursite.com/llms.txt) that tells AI assistants what the site is and which of its pages matter most. Jeremy Howard proposed it in September 2024, because full web pages, with their menus, scripts and ads, are hard for language models to read. When we checked 1,000 Y Combinator startup websites for our AI readiness study, 43% published one.
This guide covers what goes in the file, what it does and doesn't do, and how to add one.
What llms.txt is for
A web page is built for people: navigation, cookie banners, scripts and footers around the part that matters. An AI assistant reading it has to dig the facts out. llms.txt gives it the short version: who you are, what you offer, and links to the pages worth reading, in plain Markdown.
It sits beside two files you may already have, but does a different job:
| File | Written for | What it says |
|---|---|---|
| robots.txt | Crawlers | Which pages they may visit |
| sitemap.xml | Search engines | Every page you want found |
| llms.txt | AI assistants and agents | What the site is, and which pages to read |
What goes in an llms.txt file
The proposal at llmstxt.org sets out a simple order. Only the first part is required:
- An H1 with the name of the site or project.
- A blockquote with a short summary.
- Any paragraphs or lists an assistant should read before the links.
- H2 sections that list links, each written as
[name](url): a short note. - An H2 section called "Optional" for links an assistant can skip when it needs less.
Here is a short one for a made-up invoicing app:
# Acme Invoicing
> Online invoicing for freelancers and small agencies in the UK. Plans start at £9 a month.
## Product
- [Features](https://acme.example/features): Invoices, payment reminders, expenses and VAT returns
- [Pricing](https://acme.example/pricing): Every plan with its price and limits
## Help
- [Getting started](https://acme.example/docs/start): Set up an account and send a first invoice
## Optional
- [Blog](https://acme.example/blog): Guides on getting paid on timeFor a longer, real example, see ours. It's built from the same data as our pages, so a new price or feature shows up in it without anyone editing it.
How many websites have an llms.txt file?
In our study, we ran our site checker on 1,000 randomly chosen YC startup websites from the Winter 2022 to Fall 2026 batches. 43% published an llms.txt file, and newer startups were more likely to have one:
| YC batch year | Sites checked | Publish llms.txt |
|---|---|---|
| 2022 | 169 | 36% |
| 2023 | 138 | 41% |
| 2024 | 195 | 42% |
| 2025 | 219 | 44% |
| 2026 | 279 | 47% |
The share rises with every batch year, from 36% of 2022 startups to 47% of 2026 ones.
The files that often go with it are rarer. 13% of the sites also publish an llms-full.txt, 21% send a Markdown version of their home page to AI agents that ask for one, and 6% publish their prices as plain text at /pricing.md.
Does llms.txt help you show up in AI answers?
It helps any assistant that reads it, and it takes little time to make. But it isn't a ranking switch. llms.txt is a proposal rather than a web standard like robots.txt, and each AI company decides for itself whether its crawlers and assistants read it.
Three things matter more, and our study found them missing more often than llms.txt:
- Text in the HTML. 17% of the sites served a home page with fewer than 150 words of text in its HTML, and 7% sent an empty app shell that only JavaScript fills in. Most AI agents read the HTML without running JavaScript, so on those sites they find little or nothing.
- Structured data. 44% had no JSON-LD structured data on their home page, the markup that states who a business is and what it sells.
- Crawler access. Only 3 of the 1,000 sites blocked an AI search crawler in robots.txt, so it's rarely the problem. When it is, one line keeps a site out of that assistant's answers, so it's worth a check with our AI crawler checker.
Fix those first, then add llms.txt.
How to add llms.txt to your website
- Write it. Start with your name and a one-sentence summary of what you offer and who it's for. Then link the 5 to 20 pages that answer the questions buyers ask, with a short note on each. Our free llms.txt generator drafts one from your site.
- Put it at the root. It should load at yoursite.com/llms.txt as plain text. On most hosts you upload the file. On a framework such as Next.js, a route that builds it from your content keeps it up to date.
- Keep the facts current. Assistants quote what they read, including prices and plan names. Update the file when those change, or build it from the same data as your pages.
- Check it. Our free site checker looks for llms.txt alongside the other things AI assistants need to read and recommend a site.
Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which pages they may visit. llms.txt doesn't allow or block anything: it summarises the site and points to the pages worth reading.
Where does llms.txt go?
At the root of your domain, so it loads at yoursite.com/llms.txt. The proposal also allows one in a subfolder, such as /docs/llms.txt, covering the pages under that path.
What is llms-full.txt?
A companion file some sites publish with the full text of their main pages in one Markdown file, so an assistant can read everything in one request. It isn't part of the llmstxt.org proposal. 13% of the startup sites in our study had one.
How long should llms.txt be?
Short enough to read in one go. A name, a summary and 5 to 20 well-described links suit most sites. Put secondary pages under an "Optional" heading so assistants that need less can skip them.