RFC 9309 defines three directives: user-agent, allow and disallow. It lets a
crawler read other records, as long as reading them does not disturb the rules it does define. Real files carry many
more. The explorer shows every such line, labels it, and never uses it in the test of a URL. The table says what a
source we read says about each; where no source was read, the label says only that the directive is not in the RFC.
The pages were read through summaries, so the wording below is ours.
The directives
| Directive | What it is | What the source says |
|---|---|---|
sitemap | The address of a sitemap; global, not part of a group. | RFC 9309 names it as an example of an “other record” that must not end a group. The sitemaps.org protocol says it works wherever it stands in the file, and that several may be given. Google’s documentation lists it among the directives it supports. |
crawl-delay | A pause between requests. | Google’s documentation says it does not support crawl-delay. Which other crawlers read it, and how they read the number, is not sourced here. |
clean-param | Clean-param: p0[&p1&p2..] [path]: URL parameters the robot should ignore, so that addresses differing only in them count as one. | Documented by Yandex, with the example Clean-param: ref /some_dir/get_book.pl. |
host | A directive naming the main mirror of a site, from Yandex. | Yandex’s current page on robots.txt, as the summary we read gave it, does not list it. Whether Yandex still reads it is not confirmed here. |
noindex, nofollow | Rules meant to keep a page out of an index, or to stop a crawler following its links. | Google says it stopped handling unsupported and unpublished rules such as noindex on 1 September 2019. To keep a page out of search results the page itself should say so, with a robots meta tag or an HTTP header. |
request-rate, visit-time, nosnippet, noarchive, content-signal | Directives seen in files, some from the 1990s extended proposals. | No source for them was read for these pages, so the viewer labels each “not RFC 9309” and says nothing of who reads it. |
A directive nobody listed
A line such as Frobnicate: 3 is shown as an unknown directive and listed in the problems, since no source
for it was read. A line that is not name: value at all is shown as an invalid line.