llms.txtdefinition and how it works in 2026
- llms.txt
- A proposed convention (like robots.txt) for sites to tell LLMs which content to ingest, in what summary form, and on what terms. Adoption growing through 2025–2026.
llms.txt is a plain-text file at the root of a site that gives LLMs and AI agents structured guidance about the site's content. The proposal (originally from Jeremy Howard, 2024) describes a format with sections for site summary, key pages, optional content, and licensing terms — designed to be both human-readable and AI-friendly.
Unlike robots.txt (which is restrictive), llms.txt is generative — it doesn't just block crawling, it offers a curated, distilled version of the site optimized for LLM ingestion. The bet: sites that publish good llms.txt files get better AI Overviews, better citations from agents like Perplexity, and better treatment by LLM-powered search.
By mid-2026, llms.txt is on thousands of sites — including most of the major AI tool vendors, several developer-focused publishers, and aiagentrank.io itself. Whether it becomes a formal standard or stays a community convention is still open; either way, publishing one is a low-cost SEO + AI-discoverability play.
Frequently asked
Does Google or OpenAI actually respect llms.txt?+
Mixed. Anthropic and a few smaller crawlers have indicated they honor it. Google and OpenAI haven't made formal commitments as of mid-2026. The current value is more discoverability via specialized agents than guaranteed-blocking-or-allowing.
Should I publish llms.txt on my site?+
Yes if you're a publisher or product where AI discoverability matters. The cost is low (a few hundred lines of markdown), and the upside is real even before formal standardization.
Is llms.txt the same as ai.txt?+
Different proposals. llms.txt focuses on positive guidance (here's what to ingest); ai.txt focuses on restrictive guidance (here's what NOT to train on). Some sites publish both.