RFC 9309 section 2 defines the file. These notes are in our words, from the RFC read on the page of the RFC Editor and from Google’s documentation of how it reads the file.
Lines
A line ends with LF, CR LF or a lone CR. The file is UTF-8 text. A # starts a comment that runs to the end of
the line, on a line of its own or after a rule. White space (spaces and tabs) around the colon and at the ends is ignored.
The name of a directive is not case-sensitive, so User-agent, user-agent and
USER-AGENT are the same.
| Line | Form | What it does |
|---|---|---|
user-agent | user-agent: token | Opens a group, or adds a name to the group that is open. The token is a product token: letters, - and _ (RFC 9309 gives the identifier as those characters), or * for every crawler without a group of its own. |
disallow | disallow: path | A rule that a path (a pattern) may not be fetched. An empty value is a rule that excludes nothing. |
allow | allow: path | A rule that a path may be fetched; it overrides a shorter disallow. An empty value says nothing. |
sitemap | sitemap: URL | Not in RFC 9309’s grammar. RFC 9309 names it as the example of an “other record” that crawlers may read and that must not end a group. The sitemaps.org protocol defines it: the full address of a sitemap, anywhere in the file. |
| comment | # text | Ignored. |
| blank | Ignored, and it does not end a group. |
The grammar of the RFC, in its own notation:
robotstxt = *(group / emptyline)
group = startgroupline *(startgroupline / emptyline) *(rule / emptyline)
startgroupline = *WS "user-agent" *WS ":" *WS product-token EOL
rule = *WS ("allow" / "disallow") *WS ":" *WS (path-pattern / empty-pattern) EOL
product-token = identifier / "*"
path-pattern = "/" *UTF8-char-noctl
identifier = 1*(%x2D / %x41-5A / %x5F / %x61-7A)
EOL = *WS [comment] NLGroups
A group is one or more user-agent lines followed by rules. It ends at the next user-agent line
that follows a rule, or at the end of the file. User-agent lines in a row, with blank lines and comments between them,
belong to one group, and the rules after them apply to every crawler they name. The last group may have no rules at all,
which allows everything. A rule that comes before any user-agent line is in no group, and RFC 9309 says a
crawler should ignore it.
Lines of other kinds, such as sitemap, do not end a group: rules that follow one still belong to the group
before it.
Which group is for a crawler
A crawler matches its product token against the tokens of the user-agent lines, ignoring case. If one or more groups
name it, all of them are combined into one set of rules. If none does, the groups that name
* are used. If there are none of those either, no rule applies and everything is allowed. A group that
names the crawler replaces the * group for it: the * rules are not added on top.
RFC 9309 matches the token as it is. Some crawlers choose the most specific of several tokens (Google describes its own crawlers doing so); the explorer’s test follows the RFC and says so.
Lines that are not understood
RFC 9309 says a crawler must try to parse every line and use the rules it can parse. A line with no colon, a directive
nobody defined, or a rule path that does not start with / is not a rule; the viewer lists each one in the
problems and goes on. It never refuses a file because of a bad line.