What Is llms.txt? 24 Vendors Checked

Documentation sites are built for people, which means an agent that wants your API reference has to download HTML, strip navigation, ads and script tags, and hope the result still makes sense. Every wasted token there is money and latency.

llms.txt is a proposal to skip that: a single Markdown file at /llms.txt listing the documents worth reading. We checked 24 developer platforms on September 14, 2026. Twenty serve one, but only 16 at the apex domain, and Google's own documentation says Search does not use it.

What llms.txt Actually Is

llms.txt is a Markdown file placed at the root of a site that gives LLMs a curated index of the site's content, proposed by Jeremy Howard on September 3, 2024 and revised as v2 on August 10, 2026.

The specification is short. An H1 with the project name is "the only required section". Then a blockquote with a short summary, optional paragraphs, and zero or more H2 sections containing Markdown link lists. That is the whole format, which is why documentation platforms can generate one automatically.

It is not robots.txt. robots.txt tells crawlers what they may not fetch and is enforced by convention; llms.txt tells an agent what is worth fetching and enforces nothing. The two solve opposite problems and neither replaces the other.

What 24 Vendors Actually Serve

The interesting question is whether anyone is really doing this, and where the file lives. This script checks the apex domain and the documentation host for each vendor. Node 18 or newer, no dependencies.

js
Copy
// llmstxt-survey.mjs - Node 18+, no dependencies. Run: node llmstxt-survey.mjs
const SITES = [
  ["OpenAI", "openai.com", "developers.openai.com"],
  ["Anthropic", "anthropic.com", "docs.claude.com"],
  ["Stripe", "stripe.com", "docs.stripe.com"],
  ["Cloudflare", "cloudflare.com", "developers.cloudflare.com"],
  ["LangChain", "langchain.com", "python.langchain.com"],
  ["Deno", "deno.com", "docs.deno.com"],
  // ...add your own
]

async function probe(host) {
  try {
    const r = await fetch(`https://${host}/llms.txt`, { redirect: "follow", signal: AbortSignal.timeout(15000) })
    if (!r.ok) return { ok: false, status: r.status }
    const body = await r.text()
    // A 200 that returns the SPA shell is not an llms.txt.
    if (/^\s*<(!doctype|html)/i.test(body)) return { ok: false, status: "HTML" }
    return {
      ok: true, bytes: body.length,
      links: (body.match(/\]\(https?:\/\//g) || []).length,
      h1: /^#\s+\S/m.test(body), quote: /^>\s+\S/m.test(body),
    }
  } catch { return { ok: false, status: "ERR" } }
}

for (const [name, apex, docs] of SITES) {
  const a = await probe(apex)
  const d = apex === docs ? a : await probe(docs)
  const best = d.ok ? d : a
  console.log(
    `${name.padEnd(12)} apex ${(a.ok ? "yes" : String(a.status)).padStart(5)}  docs ${(d.ok ? "yes" : String(d.status)).padStart(5)}` +
      (best.ok ? `  ${String(best.bytes).padStart(7)} bytes  ${String(best.links).padStart(4)} links  ${best.h1 ? "H1" : "no H1"} / ${best.quote ? "quote" : "no quote"}` : ""),
  )
}

Run across all 24 vendors on September 14, 2026:

text
Copy
checked 24 vendors on 2026-09-14
serving /llms.txt on the apex domain:      16
serving /llms.txt on the docs host:        20
serving it somewhere:                      20

vendor          apex    docs    bytes  links  spec: H1 + blockquote
----------------------------------------------------------------------
OpenAI           403     yes    5,850     38  H1 / quote
Anthropic        404     yes   67,821    629  H1 / no quote
Google AI        404     404        -      -
Stripe           yes     yes   90,607    502  H1 / no quote
Cloudflare       yes     yes   15,902    106  H1 / quote
Vercel           yes     yes    4,721     24  H1 / quote
Next.js          yes     yes   12,710     54  H1 / quote
Svelte           yes     yes    1,674      7  H1 / quote
Zapier           yes     yes   50,532    348  H1 / quote
Fly.io           yes     yes   37,091    244  H1 / quote
Neon             yes     yes   38,548    264  H1 / quote
Deno            HTML     yes    5,053     36  H1 / quote
LangChain        404     yes   22,313    177  H1 / quote
Tailwind         404     404        -      -

bytes: min 1,674, median 15,902, max 90,607
missing the required H1: 0 (none)
missing the blockquote summary: 5 (Anthropic, Stripe, Supabase, Bun, Turso)

Three things fall out of that. The file usually lives on the documentation host, not the apex domain, so a checker that only probes openai.com/llms.txt reports a 403 and concludes OpenAI does not publish one, when developers.openai.com/llms.txt serves 5,850 bytes. Anthropic is the same, 404 at the apex and 67,821 bytes at docs.claude.com.

A 200 response is not proof of a file. deno.com/llms.txt returns HTTP 200 with the site's HTML shell, which is what a single-page app does with any unknown path. Any survey that counts status codes without inspecting the body will overcount.

Compliance is loose. All 20 have the required H1, but 5 skip the blockquote summary the spec calls for, including Anthropic and Stripe. Sizes range from Svelte's 1,674 bytes to Stripe's 90,607, and a 90 KB index is itself a meaningful chunk of a context window.

Does Anything Actually Read It?

This is where enthusiasm meets evidence. Google's own AI features documentation, last updated December 10, 2025, states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

That is Google saying the file does nothing for Search, AI Overviews or AI Mode. Google Search Advocate John Mueller has been blunter in public, comparing the idea to the meta keywords tag and noting server logs showed no AI system fetching it.

The honest read is that llms.txt has one real constituency and it is not search engines. It is coding agents and chat assistants that fetch documentation on demand, where a curated index genuinely saves tokens. That is exactly the population the spec's own background section describes, and exactly why the vendors publishing one are documentation-heavy developer platforms rather than general websites.

Should You Publish One?

1. Publish it if agents read your docs. If you ship an SDK, an API or a framework, a curated index measurably reduces what an agent has to fetch. That is the case all 20 vendors above are in.
2. Do not publish it for SEO. Google has said in writing that it does not use the file. Treat any claim that llms.txt improves rankings as unevidenced.
3. Put it where your docs are. Serve it from the documentation host, and from the apex too if you can, since tools check both inconsistently.
4. Return text/plain or text/markdown, never HTML. Test with curl -s https://yoursite.com/llms.txt | head -1. If the first line starts with <, your SPA is intercepting the route and every agent sees the shell.
5. Follow the two-element minimum. An H1 and a blockquote summary cost one line each, and a quarter of the files we checked skipped the blockquote.
6. Keep it an index, not a dump. The median here is about 16 KB. If yours approaches Stripe's 90 KB, the file has stopped being a cheap pointer and become something an agent must budget for.

Conclusion

llms.txt is a genuinely useful convention for one specific job: telling a coding agent which of your documentation pages are worth fetching. It is not a ranking signal, Google has said so in its own docs, and a survey of 24 vendors shows even the sites that publish one treat the spec loosely. If you maintain developer documentation, spend the hour. If you were hoping it would move you in AI search results, spend the hour on the content instead.

Related DevToolLab Tools

• llms.txt Generator - build a spec-shaped file with the required H1 and blockquote from your existing page list instead of hand-writing Markdown.
• AI Crawler Blocker - the other half of the policy question, generating the robots.txt rules that decide which AI crawlers may fetch you at all.
• robots.txt Tester - check that the crawler rules you think you shipped are the ones actually being served.
• Sitemap Validator - the machine-readable index search engines genuinely do use, worth fixing before adding one they do not.

Related Guides

• Generative Engine Optimization Guide - what actually influences citation in AI answers, with llms.txt in its proper place.
• What Is AGENTS.md? Inside 24 Real Repos - the same idea aimed at coding agents working inside a repository rather than reading a website.
• Best AI Code Documentation Generators - the tooling that produces the pages an llms.txt would point at.
• Best MCP Servers - the other route agents use to reach your data, with a protocol instead of a text file.