robots.txt Tester
Fetch a domain's robots.txt, view all rules, and test whether a URL is allowed or blocked.
About robots.txt Tester
Tests whether a specific URL is allowed or disallowed under a domain's robots.txt rules, evaluating the actual directive precedence — including wildcard patterns and the most-specific-rule-wins logic — rather than requiring you to trace through the file by hand. Robots.txt syntax looks simple but the precedence rules between overlapping Allow and Disallow directives are easy to get wrong, and a misconfigured rule can accidentally block search engines from content you wanted indexed. Worth testing after any robots.txt change.
Disallowing a URL in robots.txt stops crawling, not indexing — a blocked page linked from elsewhere can still show up in search results (usually with no snippet); a noindex meta tag is the actual tool for fully hiding a page.
Related Tools
How robots.txt rules are matched
The rules are standardized in RFC 9309. A crawler first picks the group whose User-agent line matches it most specifically, and falls back to User-agent: * only if none does. Inside that group:
- The longest matching path wins, whether it is an Allow or a Disallow.
- If an Allow and a Disallow match with the same length, Allow wins.
*matches any sequence of characters, and$anchors the end of the URL.- Paths are case-sensitive, and an empty
Disallow:allows everything.
| Rules | URL | Result |
|---|---|---|
Disallow: /adminAllow: /admin/public | /admin/public/page | Allowed (longer match) |
Disallow: /*.pdf$ | /files/report.pdf | Blocked |
Disallow: /*.pdf$ | /files/report.pdf?v=2 | Allowed (does not end in .pdf) |
Mistakes that hurt search traffic
Disallow: /left over from a staging site. It blocks the whole site.- Blocking CSS and JavaScript. Google renders pages, and blocked assets can make a page look broken to it.
- A robots.txt that returns a server error. Google treats a 5xx response as a temporary block of the whole site. A 404, by contrast, means everything is allowed.
- Using robots.txt to hide a page. A blocked page can still be indexed from links. Use
noindexinstead, and don't block that page in robots.txt, or crawlers will never see the noindex.
What this tool shows
It fetches the domain's robots.txt, lists every user-agent group and the declared sitemaps, and tests a URL (path and query string) with the RFC 9309 rules above, wildcards included. Enter a crawler name such as Googlebot to test the group that crawler would use; leave it empty to test User-agent: *. The result shows the matching rule and the group that was applied. A robots.txt that returns 404 means no restrictions.