Rules are matched against the path of a URL, with its query string: /fish/salmon.html?id=3. RFC 9309
section 2.2.2 gives the rules; the examples in the tables are the ones the RFC and Google’s documentation give, and the
explorer’s tests check each of them.
The matching rules
- A rule matches when the end of its pattern is reached before a difference is found: the pattern is a prefix of the path.
/fishmatches/fish,/fish.htmland/fishheads/yummy.html. - The comparison is case-sensitive, so
/fishdoes not match/Fish.asp. It starts at the first octet of the path. *stands for any number of any characters, including none.$at the end of a pattern means the path must end there.- When several rules match, the one with the longest pattern (the most octets) wins. If an allow and a disallow match with the same length, the allow wins. If no rule matches, the path is allowed.
- The address
/robots.txtitself is always allowed.
What a pattern matches
| Pattern | Matches | Does not match |
|---|---|---|
/fish | /fish, /fish.html, /fish/salmon.html, /fishheads, /fish.php?id=anything | /Fish.asp, /catfish, /?id=fish, /desert/fish |
/fish* | the same as /fish: a trailing * adds nothing | the same as /fish |
/fish/ | /fish/, /fish/?id=anything, /fish/salmon.htm | /fish, /fish.html, /Fish/Salmon.asp |
/*.php | /filename.php, /folder/filename.php?parameters, /folder/any.php.file.html, /filename.php/ | /, /windows.PHP |
/*.php$ | /filename.php, /folder/filename.php | /filename.php?parameters, /filename.php/, /filename.php5, /windows.PHP |
/fish*.php | /fish.php, /fishheads/catfish.php?parameters | /Fish.PHP |
/ | every path | |
/$ | only / | every other path |
Which rule wins
| Rules | URL path | Result |
|---|---|---|
allow: /p and disallow: / | /page | allowed: the allow is longer |
allow: /folder and disallow: /folder | /folder/page | allowed: equal length, allow wins |
allow: /$ and disallow: / | / | allowed |
allow: /$ and disallow: / | /page.htm | disallowed: only the disallow matches |
allow: /page and disallow: /*.htm | /page.htm | the RFC calls this undefined; the explorer counts the characters of the pattern, * included, so the longer disallow wins |
The order of the rules in the file does not matter. Python’s urllib.robotparser works differently: it
takes the first rule that matches, in file order, and does not read * or $ in a path. A
file written for one reading can give another answer under the other.
Percent-encoding
Before comparing, both the path and the pattern are normalised. A character outside ASCII is written as its UTF-8
octets, each as %XX, so a rule with the literal character ツ matches a URL that has
%E3%83%84 and the other way round. A percent-encoded ASCII letter, digit or one of - . _ ~
is decoded, so /foo/bar/%62%61%7A matches the pattern /foo/bar/baz. An encoded reserved
character such as %2F stays encoded and is not the same as the character itself.