Size
RFC 9309 section 2.5 says a crawler’s parsing limit must be at least 500 kibibytes (512,000 bytes), and that a crawler should impose one. Google documents a limit of 500 KiB and says that content after it is ignored. The explorer applies 512,000 bytes: lines after that offset are shown as one ignored block, and the line that crosses it is cut. Rules written after the limit do nothing, so a long file should put what matters first.
Encoding, byte order mark and line ends
- The file is UTF-8 plain text (RFC 9309; Google says the same). A file in another encoding is shown, with a problem that says so; a UTF-16 file with a byte order mark is read as UTF-16.
- A UTF-8 byte order mark at the start is not mentioned by the RFC. Google documents that it ignores one. The viewer gives it its own node and does not count it as a line.
- A line ends with LF, CR LF or a lone CR. A file may mix them; the viewer notes it.
When the file is fetched
These rules are about the HTTP request for /robots.txt, not about the file; the viewer states them as text
on the Overview and never fetches anything. Section numbers are those of RFC 9309.
| Answer | What a crawler does |
|---|---|
| 2xx (2.3.1.1) | Follows the rules it can parse. |
| 3xx (2.3.1.2) | Should follow at least five consecutive redirects, even to another host; the rules then apply to the host first asked. After more than it follows, it may treat the file as unavailable. Google documents following at least five hops and then treating the file as a 404. |
| 4xx (2.3.1.3) | The file is unavailable, and the crawler may fetch anything. Google treats these as no robots.txt, except 429, and warns against using 401 or 403 to limit crawling. |
| 5xx and network errors (2.3.1.4) | The file is unreachable: the crawler must assume everything is disallowed. Google documents stopping for 12 hours, then using its last good copy for up to 30 days. |
A crawler should not use a cached copy for more than 24 hours unless it cannot reach the file (2.4). Rules apply only to the host, protocol and port that served the file; a subdomain needs its own.