Introduction
llms.txt is a proposal by Jeremy Howard (llmstxt.org, version 2, header dated
3 September 2024) for a Markdown file that gives a language model or agent a short, curated guide to a site: what it is and
where the useful pages are. It is not a standard of a standards body, and the pages here describe the proposal as it is
written. It sits at the site root, /llms.txt, or at any path below it, in which case it covers the pages under that path.
Structure
The file is Markdown with a fixed order. Only the first part is required.
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)
- An optional byte-order mark.
- An H1 with the name of the project or site. “This is the only required section.”
- A blockquote with a short summary.
- Zero or more Markdown sections of any type except headings (paragraphs, lists…).
- Zero or more sections delimited by H2 headings, each a list of links. Each item has a required Markdown link
[name](url)and optionally a colon and notes.
The section named Optional is by convention for secondary links an agent can skip when a shorter context is needed.
The proposal also suggests that pages offer a Markdown copy at the same address with .md added, and links such as
rel="alternate" type="text/markdown" and rel="describedby"; those are not part of the file.
In the explorer
Drop a file named llms.txt on the explorer to see it as the sections the proposal names: the title with its summary
and details, then one group for each H2 section with a node for each link. The tabs list the sections, every link with its section,
URL and notes, and the problems. Lines that the proposal does not describe are listed as Issues, hedged as such, and never refuse
the file: no H1 at the start, text before the H1, a second H1, a heading other than H2 in the file lists, a list item with no link, text
after a link that does not start with a colon, an empty link name or URL, a blockquote out of place and an unclosed code fence. Lines
inside code fences are not read as links. A link is read as CommonMark reads an inline link: a URL may hold balanced parentheses
(https://en.wikipedia.org/wiki/Foo_(bar)), may be written in angle brackets ([a](<https://x.y/a b>)), and may carry a
title; a link wrapped in bold (**[a](url)**: notes) is a link with notes. A title underlined with === (a setext heading) is read
as the H1, and one underlined with three or more - as an H2. A table of more than 5,000 rows is paged, 200 at a time with Previous
and Next, and the Overview gives the total: no link is left out.
A file is recognised as llms.txt by its name and its content. By name (how a dropped or listed llms.txt reaches the
viewer) it is enough that the text is not HTML or XML and shows the structure of the format anywhere: a blockquote, an ## heading or a list item
with a Markdown link outside code fences, or is a single H1 alone. So a file with no H1, or with text before the H1, opens in the viewer and
says so as an Issue. The stricter content test (the first line with content is an H1, and the file shows a blockquote, an H2 or a link) is kept for
callers that have no name; a file of a title and ordinary paragraphs, prose and a text with none of these is plain text. Any other file called
llms.txt opens as plain text. There is no size limit on the claim (the viewer says "too large" above 8 MiB). The Text tab is not
coloured by Markdown rules here.
Not described
Known, not yet covered
- Viewer: only the headings, blockquote and list items that the proposal names are interpreted; the rest of the Markdown (emphasis, tables, nested lists) is shown as text. The explorer’s Markdown viewer is not used for the Text tab.
- Viewer: the link reader handles the inline form
[name](url "title")only: reference-style links ([name][ref]), autolinks (<https://…>) and a link whose text spans two lines are not read as links (CommonMark's rules were applied from memory of the CommonMark specification, which was not fetched for this work: our reading). A setext heading is read for a single title line only. - Viewer: a file called llms.txt that has none of a blockquote, an H2 or a list link (a title with paragraphs, or a lone H2 with text) is not recognised and opens as plain text, so its missing H1 or other faults are not shown.
- Reference: the sibling convention
llms-full.txt(all the documentation in one file) is used by some sites but is not in the specification text read for this page, and is not recognised as a format here.
Unknown
- Reference: a size limit for the file, the maximum number of links, or the exact Markdown dialect (CommonMark or other): the proposal states none.
- Viewer: how a given agent treats a list item without a link or a heading that the proposal does not describe: the proposal does not say.
Links
Reference
- The /llms.txt file – the proposal, with its example
- AnswerDotAI/llms-txt – the repository of the proposal and its sample code