robots.txt Tester

Tests whether a specific URL is allowed or disallowed under a domain's robots.txt rules, evaluating the actual directive precedence — including wildcard patterns and the most-specific-rule-wins logic — rather than requiring you to trace through the file by hand. Robots.txt syntax looks simple but the precedence rules between overlapping Allow and Disallow directives are easy to get wrong, and a misconfigured rule can accidentally block search engines from content you wanted indexed. Worth testing after any robots.txt change.

Disallowing a URL in robots.txt stops crawling, not indexing — a blocked page linked from elsewhere can still show up in search results (usually with no snippet); a noindex meta tag is the actual tool for fully hiding a page.

📖 Read more in our Knowledge Base →