robots.txt File Format

Introduction

A robots.txt file tells web crawlers which parts of a site they may fetch. It sits at one fixed address, /robots.txt at the top of the site, and is plain text. The convention dates from the 1990s and was written down as a standard in 2022, as RFC 9309, the Robots Exclusion Protocol.

It is a request, not a lock: a well-behaved crawler reads the file before it fetches anything else from the host and obeys it, and nothing in the protocol stops one that does not. A page the file disallows can still be listed by a search engine if other pages link to it; Google says so in its documentation.

Structure

The file is lines of directive: value, with comments after #. The lines fall into groups. A group starts with one or more user-agent lines, which name the crawlers it is for, and goes on with allow and disallow rules, which name paths. A sitemap line is not part of a group: it points to a sitemap and applies whatever group it is written in.

# Everyone: stay out of the private area, except one page in it
User-agent: *
Disallow: /private/
Allow: /private/press-kit.html

# One crawler by name, and a second one that gets the same rules
User-agent: ExampleBot
User-agent: OtherBot
Disallow: /*.php$

Sitemap: https://example.com/sitemap.xml

To decide whether a URL may be fetched, a crawler picks the group that names it, or the * group, and takes the rule whose path is the longest match for the URL; when an allow and a disallow match equally, allow wins. Matching has the rules for *, $ and percent-encoding. A file with no rule that matches allows everything.

In the explorer

Drop a file named robots.txt on the explorer to see it as groups and lines: each group with its user-agent lines and rules, the sitemap lines, comments, and any line that is not understood, marked as a problem. The tabs list the groups, every rule, the sitemaps and the directives that RFC 9309 does not define, each labelled and never dropped.

The options bar has a Test a URL box and a User-agent list made from the file’s own user-agent lines. Type a path such as /private/page.html, or a whole URL, and the Overview shows which group applies, every rule that matches and why one of them wins, and whether the URL is allowed.

A file is recognised as robots.txt by its name and by its content: at least one directive line, and four of every five lines that are not comments are directive lines. Any other file called robots.txt (a page of prose, an error page, an empty file) opens as plain text, as it always did. Until the highlighting for this format is added the text is shown without colours.

Not described

Known, not yet covered

• Viewer: the file is shown as plain text, without colours, until its highlighting is added to the explorer.
• Viewer: a crawler that matches a product token by prefix (Google’s documentation describes its own crawlers choosing the most specific group) is not modelled: the test matches a token exactly, as RFC 9309 says, and falls back to the * group.
• Reference: the directives Bing, Yandex and others read beyond RFC 9309 are listed on Other directives only where a documentation page was read; the rest are labelled as not in the RFC and nothing more.
• Viewer: the test uses the rules of RFC 9309. Where Google’s published parser or Python’s urllib.robotparser differ from it (rule order, wildcards) the result follows the RFC.

Unknown

• Reference: how each search engine treats a malformed line (a misspelt directive, a missing colon) is not described in the documentation we read. The viewer reports such a line and ignores it, as the RFC permits.

Links