Robots.txt Tester and Validator

RFC 9309 and Google's documentation describe the same four steps, and the tester follows them exactly:

1. Pick the group. Find the group whose User-agent equals the crawler's product token, ignoring case. Several groups for the same token are merged. If none matches, use the * group; if there is no * group either, everything is allowed.
2. Match the path. Compare each rule with the URL's path and query, from the first character. Paths are case-sensitive.
3. Longest rule wins. The most specific match, the rule with the most characters, decides.
4. Allow wins a tie. When an Allow and a Disallow match with equal length, the URL is allowed.

With the file below, Googlebot may fetch /admin/settings (its own group has no rule for it) but not /drafts/post, while any other crawler may open /admin/help/start (Allow: /admin/help/ is longer than Disallow: /admin/) but not /admin/settings:

User-agent: *
Disallow: /admin/
Allow: /admin/help/

User-agent: Googlebot
Disallow: /drafts/

/robots.txt itself is always allowed, and an empty Disallow: allows everything.

Because only one group applies to a crawler. Once a group names Googlebot, the * group no longer applies to it: Google says groups for a specific user agent and the global group are never combined. It is the most common surprise in a robots.txt audit. If Googlebot should follow your general rules too, repeat them in its group.

Some crawlers fall back to another group before *. Google lists Googlebot-Image with two tokens, so with no group of its own it obeys Googlebot's. Apple says Applebot follows Googlebot's rules when robots.txt doesn't mention Applebot. The tester applies these documented fallbacks when you pick those crawlers.

Google also reads only the product token: User-agent: Googlebot/2.1 and User-agent: googlebot* both mean googlebot. User-agent lines don't support wildcards, so User-agent: *bot matches no crawler at all.

* matches any run of characters, including none, and a $ at the end anchors the rule to the end of the URL. Everything else is a plain prefix match:

A trailing * changes nothing: /shop* matches the same URLs as /shop. Use $ with care: /*.pdf$ lets /menu.pdf?download=1 through, because the query string is part of what is matched.

RFC 9309 compares rules and URLs in the same percent-encoded form. The tester normalizes both before matching:

• Non-ASCII characters are encoded as UTF-8, so Disallow: /café/ and Disallow: /caf%C3%A9/ are the same rule.
• Encoded letters, digits and - . _ ~ are decoded, so /%62log/ matches a rule for /blog/.
• Encoded reserved characters stay encoded: %2F is not a path separator, so /a%2Fb doesn't match /a/b.
• To match a literal * or $ in a URL, write it as %2A or %24 in the rule.

When a URL is compared in a different form than you typed it, the detail panel shows the normalized path.

Search Console no longer has its old robots.txt Tester. Its robots.txt report shows which files Google found for your top 20 hosts, when it last crawled them, and any warnings or errors, and it can request a recrawl after an urgent fix. To check one live URL, the URL Inspection tool says whether crawling is allowed.

Neither lets you try a change before you publish it or compare crawlers, which is what this tester is for: paste a draft, test the URLs that matter, then upload. To write a new file from scratch, start with the robots.txt generator.