Testers

robots.txt Tester

Fetch a domain's robots.txt, view all rules, and test whether a URL is allowed or blocked.

Raw robots.txt

About robots.txt Tester

Tests whether a specific URL is allowed or disallowed under a domain's robots.txt rules, evaluating the actual directive precedence — including wildcard patterns and the most-specific-rule-wins logic — rather than requiring you to trace through the file by hand. Robots.txt syntax looks simple but the precedence rules between overlapping Allow and Disallow directives are easy to get wrong, and a misconfigured rule can accidentally block search engines from content you wanted indexed. Worth testing after any robots.txt change.

Disallowing a URL in robots.txt stops crawling, not indexing — a blocked page linked from elsewhere can still show up in search results (usually with no snippet); a noindex meta tag is the actual tool for fully hiding a page.

📖 Read more in our Knowledge Base →

How robots.txt rules are matched

The rules are standardized in RFC 9309. A crawler first picks the group whose User-agent line matches it most specifically, and falls back to User-agent: * only if none does. Inside that group:

  • The longest matching path wins, whether it is an Allow or a Disallow.
  • If an Allow and a Disallow match with the same length, Allow wins.
  • * matches any sequence of characters, and $ anchors the end of the URL.
  • Paths are case-sensitive, and an empty Disallow: allows everything.
RulesURLResult
Disallow: /admin
Allow: /admin/public
/admin/public/pageAllowed (longer match)
Disallow: /*.pdf$/files/report.pdfBlocked
Disallow: /*.pdf$/files/report.pdf?v=2Allowed (does not end in .pdf)

Mistakes that hurt search traffic

  • Disallow: / left over from a staging site. It blocks the whole site.
  • Blocking CSS and JavaScript. Google renders pages, and blocked assets can make a page look broken to it.
  • A robots.txt that returns a server error. Google treats a 5xx response as a temporary block of the whole site. A 404, by contrast, means everything is allowed.
  • Using robots.txt to hide a page. A blocked page can still be indexed from links. Use noindex instead, and don't block that page in robots.txt, or crawlers will never see the noindex.

What this tool shows

It fetches the domain's robots.txt, lists every user-agent group and the declared sitemaps, and tests a URL (path and query string) with the RFC 9309 rules above, wildcards included. Enter a crawler name such as Googlebot to test the group that crawler would use; leave it empty to test User-agent: *. The result shows the matching rule and the group that was applied. A robots.txt that returns 404 means no restrictions.