Robots.txt Tester
Check whether a URL is allowed or blocked by robots.txt.
No matching rule — crawling is allowed by default.
The most specific user-agent group is chosen, then the longest matching path rule wins; on ties, Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported.
Free robots.txt tester. Paste your robots.txt, enter a URL path and a user-agent, and instantly see whether crawling is allowed or blocked — with the exact matching rule. Supports wildcards and end-anchors.
How to use the Robots.txt Tester
- 01Paste your robots.txt contents.
- 02Enter the URL or path you want to test.
- 03Enter the crawler's user-agent (e.g. Googlebot).
- 04See whether it is allowed or blocked and which rule matched.
Frequently asked questions
How does the tester decide allowed vs blocked?
It follows the standard rules: the most specific user-agent group applies, then the longest matching path rule wins, and on ties an Allow beats a Disallow. Wildcards (*) and end-anchors ($) are supported.
Does it fetch my live robots.txt?
No — you paste the contents, so nothing leaves your browser. That also lets you test changes before you publish them.