What is llms.txt, and does it help AI search? (2026)

llms.txt is a Markdown file at a website's root that gives AI tools a short summary of the site and links to its best pages. It doesn't help Google rankings or AI Overviews, because Google says Search ignores it. No major AI platform has said it reads the file to answer chat questions either.

Yet OpenAI, Anthropic and Google all publish one. Each covers developer docs. That's the clue. The file has a real job, which is helping AI agents read a site quickly, and ranking has nothing to do with it.

I'm Brenden House, and I run Searchmaxxed, an SEO and AI search agency for enterprise, B2B SaaS, venture-backed, manufacturing and public-sector companies. This guide is for the marketing and engineering leads deciding whether to ship one. Here's what it is, who reads it and what to fix first.

In summary

llms.txt is a map of your site for AI agents, and it won't move your Google rankings or get you named in ChatGPT.

• Fix crawl access and indexing first. They decide whether Google and ChatGPT search can use your pages at all, file or no file.
• If your CMS or docs platform offers one, switch it on and read every line, because generators list pages you forgot about.
• Judge AI visibility by the answers and your server logs, never by whether a file exists.

What is llms.txt?

llms.txt is a proposed standard for one Markdown file that tells large language models what a site is and where its best pages live. Jeremy Howard of Answer.AI published the first proposal in September 2024. Version 2 arrived in August 2026.

The problem it solves is real. Site owners design pages for people, so the facts sit inside menus, scripts, cookie banners and ads. An agent must strip all that away first. The proposal says converting HTML "back into clean text is difficult and imprecise", and every wasted token costs time and money. A short Markdown file skips the mess.

A large language model (LLM) is the AI system behind tools like ChatGPT, Claude and Gemini. A token is a chunk of text. Fewer tokens of page junk means a faster, cheaper and more accurate read.

The file follows a set order:

1. An H1 with the name of the site, which is the only required part.
2. A blockquote with a one-line summary of what you do and who for.
3. Optional plain paragraphs with notes an agent should know.
4. H2 sections holding lists of links, written [name](url), with an optional note after a colon.
5. An "Optional" section for links an agent can skip when it's short on room.

What changed in llms.txt version 2?

Version 2 is mostly about discovery. Each page can point to its Markdown copy with rel="alternate" type="text/markdown". It can also point to its llms.txt with rel="describedby", in HTML or a Link header.

It settled two loose ends as well. A Markdown copy can live at page.html.md or page.md. A file in a subfolder, such as /docs/llms.txt, covers only the pages under that folder, and the most specific file wins.

What does an llms.txt file look like?

It looks like a short Markdown page: a title, a one-line summary and lists of links. Here's a made-up one for a B2B software company.

# Example Payroll

> Payroll and Single Touch Payroll software for Australian businesses with 20 to 2,000 staff.

Prices on the site exclude GST. Award interpretation covers modern awards only, not enterprise agreements.

## Product

- [Features](https://www.examplepayroll.com.au/features.md): pay runs, STP Phase 2, leave, super and award interpretation
- [Pricing](https://www.examplepayroll.com.au/pricing.md): plans, per-employee fees and contract terms
- [Security](https://www.examplepayroll.com.au/security.md): SOC 2, ISO 27001, Australian data hosting and access controls

## Docs

- [API reference](https://www.examplepayroll.com.au/docs/api.md): endpoints, auth and rate limits
- [Xero integration](https://www.examplepayroll.com.au/docs/xero.md): setup and sync rules

## Optional

- [Blog](https://www.examplepayroll.com.au/blog.md)

That's the whole idea: a curated sitemap with a summary on top.

What is llms-full.txt?

llms-full.txt is the companion file. It puts the full text of your key pages into one Markdown document, ready to grab in a single request. Ours runs to about 490 KB. That's why the short index exists: no agent wants half a megabyte to find one price.

Does llms.txt help SEO?

No. Google's guide to AI features in Search lists llms.txt among the things you can ignore, for AI Overviews, AI Mode and normal results alike.

Google's guidance is blunt. You "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities)". The reason: "Google Search itself doesn't use them."

Google may still crawl and index the file like any other text file, and you might even spot it in a site: search. Your rankings won't change.

Google's John Mueller compared it to the old keywords meta tag, as Ahrefs reports, calling it "what a site-owner claims their site is about". If a model can check the real page, why trust the owner's summary?

Google's own bar is low. Your page must be indexed and eligible to show with a snippet, and Google's AI features page adds: "There are no additional technical requirements". That's ordinary technical SEO. llms.txt plays no part in it.

Does llms.txt help you show up in AI Overviews?

No. AI Overviews draw on Google's normal index. Google says it doesn't use llms.txt, so the winning page is one Google has indexed and trusts, file or no file. Our guide to ranking in ChatGPT covers the other half: getting named once an AI answer appears.

Does ChatGPT use llms.txt?

Not that anyone has shown. No major AI platform, OpenAI included, has said it reads llms.txt to answer questions. The published tests come up empty.

The AI companies' own files point the same way. The proposal links to the llms.txt files OpenAI, Anthropic and Google publish, and all three cover only developer documentation. That tells you who reads them.

So who does read llms.txt?

Agents with a job read it. A coding assistant fetching an API reference will follow an llms.txt straight to the right page. The version 2 notes say coding agents now "use them reliably". A research agent shortlisting software for a buyer can use one to reach pricing, security and integration pages without crawling the whole site.

Chat answers are a different matter. When ChatGPT answers a question, it searches the web and reads pages, and on 2 October 2026 we measured how picky it is:

We put 300 Australian agency-buyer questions to ChatGPT three times each, with search on. It pulled 17,870 search results into those 900 answers and kept 4,389 as sources. Being found is the easy part. About three in four results it reads never make the answer.

A tidy llms.txt doesn't change that cut. What ChatGPT keeps depends on what the page says and who else vouches for you.

How many websites have an llms.txt?

About one in ten. SE Ranking checked nearly 300,000 domains and found the file on 10.13% of them.

Big sites aren't ahead, either. In the same study, sites with more than 100,000 visits adopted it at 8.27%, slightly below mid-traffic sites at 10.54%. For most companies, it's still optional.

Should your company add llms.txt?

Add it if AI agents need to find things on your site, or if your platform makes it for free. If someone would have to write and update it by hand for a small marketing site, it can wait for now.

The risk is small but real. A generated file can list draft pages, retired offers or pages you deleted last year. Read yours the way an agent would. Then cut anything your sales team wouldn't say to a prospect.

How should a large site structure llms.txt?

Split it. One file with thousands of links won't fit in an agent's context window, so it starts searching the file for keywords and misses things.

Cloudflare plans around this for its own documentation, which runs past 5,000 pages. Its fix was a separate llms.txt for each top-level product folder, with the root file pointing to those. It also dropped directory-listing pages that only repeat links, and gave every entry a clear title, a matching URL and a short description.

Then Cloudflare tested it on technical questions. Its agent used 31% fewer tokens and reached the right answer 66% faster than on the average docs site not refined for agents. Version 2's subfolder rule supports that structure, so a /docs/llms.txt or /products/llms.txt covers just its own section.

How do you create an llms.txt file?

Write a short Markdown file, publish it at your site root and check it loads as plain text. A small site takes under an hour. On a large one, generate it from your CMS or product data on every release.

1. List the pages that answer real questions. Products, pricing models, security, integrations, policies, case studies, contact and your best guides. Skip tag pages, logins and search results.
2. Write the summary. One sentence on who you are, what you sell and who for. Match the wording on your homepage and LinkedIn page.
3. Group the links under H2 headings, each with a short note after a colon.
4. Publish it so it loads at yourdomain.com/llms.txt.
5. Check it in a browser. You should see plain text, not a styled page.
6. Keep it current. Regenerate it on every release that adds or retires a product, page or price.

Which llms.txt generator should you use?

Use whatever already runs your site. On WordPress, Rank Math, Yoast SEO and All in One SEO can each generate the file. Documentation platforms such as Mintlify and GitBook generate one automatically.

Free online generators crawl your site and hand you a draft to edit, because a crawler can't tell your best page from your worst. On a large site, copy our approach. Build the file from the same data that builds your sitemap, so the two can never drift apart.

What is the difference between llms.txt, robots.txt and sitemap.xml?

robots.txt controls which crawlers may visit. sitemap.xml lists the pages you want indexed. llms.txt only guides AI agents, and it can't grant or block anything, whatever you write in it.

robots.txt is the file that matters for AI search. OpenAI says a site can allow OAI-SearchBot "in order to appear in search results" while blocking GPTBot from training. Block OAI-SearchBot in robots.txt or at your firewall, and ChatGPT search can't fetch your pages. Your llms.txt won't save you.

Firewalls catch big sites out too. A bot-protection rule set by the security team can turn AI crawlers away with a 403 error, and marketing may never know. Check your CDN's bot settings as well as the text file.

What did we set up on searchmaxxed.com?

We run the full version 2 setup, and I treat it as housekeeping. Our site builds llms.txt and llms-full.txt from the same route list that builds the sitemap, so both stay current with each release.

Our llms.txt opens with a one-line summary and a note that packages are quoted for each business. Then it lists more than 260 pages. Each link goes to that page's Markdown copy, and every HTML page sends a Link header pointing agents back to llms.txt. Our robots.txt names the AI search and training crawlers and allows both.

The full checklist, with what each item saves, is in our guide to what makes a website agent-ready. Do I think any of it is why people find us in ChatGPT? No. It makes our pages cheap to read, but the page and the sources around it decide whether an AI answer picks us.

How can you tell if AI tools use your llms.txt?

Check your server or CDN logs for requests to /llms.txt and note the user agent. That's the only direct evidence.

Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Google-Extended. Then compare their hits on llms.txt with their hits on your real pages. A crawler that takes your product pages daily and never touches llms.txt has answered the question for you.

Be careful with AI referral numbers in GA4, because bots can pose as assistant traffic, as we saw on an enterprise B2B client's account:

A client's GA4 showed AI-assistant visits jump from 354 to 6,050 in a month. Almost none of it was real. About 5,700 came in three one-hour bursts: same 1920x1080 desktop Chrome, all new users, no referrer, about 20 seconds each, landing on two pages. Real AI traffic was 385 sessions against 354 the month before, roughly flat.

Picture shipping an llms.txt that month and reading that report without checking the hours, devices and referrers. It would look like magic. For Google's AI features, use the Generative AI performance report in Search Console. To track what ChatGPT, Gemini and Perplexity say about you, compare the options in our AI visibility tools guide.

What gets a company into AI answers instead?

The same work that earns Google rankings, plus a reputation on other sites. AI Overviews draw on Google's index. ChatGPT needs OAI-SearchBot to reach your pages before it can quote them.

• Crawl access. Let Googlebot, Bingbot and OAI-SearchBot through robots.txt and your firewall.
• Indexed pages with clean text. Key facts belong in the HTML, not inside images or scripts that run after load.
• Pages that answer the question. One page per product, use case, integration and market, with the answer in the first lines.
• Consistent facts. The same name, certifications, regions and pricing model on your site and everywhere else.
• Mentions on the pages AI already trusts. Comparison lists, review sites, analyst coverage and press.

That last point carries the most weight, and our ChatGPT test showed it plainly:

Across 900 ChatGPT answers about Australian agencies, a single third-party "best agencies" site was cited in 168 of them. The next three were a second listicle, Semrush's agency directory and Clutch. If you want to be recommended, the job is getting onto the handful of pages the model already trusts.

No file on your own server reaches those pages, and our guide to AI SEO explains how the on-site and off-site work fit together.

Where does Searchmaxxed fit?

Searchmaxxed is an SEO and AI search agency for enterprise, B2B SaaS, venture-backed, manufacturing and public-sector companies. We do technical SEO, content, link building, digital PR, GEO, AEO and LLM SEO. The aim: rank in Google and get named in ChatGPT, Gemini, Perplexity and Google's AI answers for the searches that bring pipeline.

We fix crawl access, indexing, rendering, speed, schema, internal links and AI crawler rules. Where an agent-readable layer helps, such as llms.txt and Markdown copies of key pages, we add it as housekeeping inside the technical work.

Here's how we run it. We build on a website you own, our Agentic Website. AI agents pull your Search Console and GA4 data, read the Google results and check what AI answers say about you and your competitors. That feeds the Managed Search Loop, which shows which page, fix or link should bring in the most enquiries next. Off your site, the Off-Page Source Layer builds the links, digital PR and mentions AI answers draw on.

The managed plans differ in volume:

• Growth System. 30 page releases a month, work on up to four existing pages and AI visibility tracking.
• Agentic Search Engineering. 50 page releases a month, comparison and industry pages, digital PR and AI answer tracking across ChatGPT, Gemini, Perplexity and AI Overviews.
• Custom & Enterprise. Several markets, dedicated engineering, custom reporting and original research.

To start with a diagnosis, the AI SEO Audit takes five business days and ranks every fix by what it's worth.

Before Searchmaxxed, I built Radiant Roof Repairs from zero to more than A$1 million a year in revenue using SEO as the only marketing channel. Three high-intent Gold Coast roofing searches reached number one within 90 days. I did the pages, research, competitor checks and updates manually.

We now run the same method for larger companies, with AI agents doing much of the legwork.

Searchmaxxed's work has generated more than A$10 million in revenue through search for its clients.

Is Searchmaxxed right for you?

We suit marketing and growth teams that want Google and AI search from one team, on a site they own, with monthly releases. If you only need an llms.txt, you don't need us.

Frequently asked questions

What should you do next?

Open yourdomain.com/llms.txt and your robots.txt today. If AI crawlers are blocked or your key pages aren't indexed, fix that first; our technical SEO service does exactly that work. If you'd like us to scope it for your site, let's talk pricing.