FAHAQ

Robots.txt Tester

Check whether a URL is allowed or blocked by robots.txt.

Allowed

No matching rule — crawling is allowed by default.

The most specific user-agent group is chosen, then the longest matching path rule wins; on ties, Allow beats Disallow. Wildcards (*) and end-anchors ($) are supported.

ShareXFacebookLinkedInWhatsAppRedditEmail

Free robots.txt tester. Paste your robots.txt, enter a URL path and a user-agent, and instantly see whether crawling is allowed or blocked — with the exact matching rule. Supports wildcards and end-anchors.

What a robots.txt tester does

A robots.txt tester takes your rules and a URL, then tells you whether a given crawler (e.g. Googlebot) would be allowed or blocked from fetching that URL. Because matching precedence isn't obvious — Allow and Disallow interact, and the most specific rule wins — testing beats guessing.

Use it before deploying changes and whenever pages you expected in Google mysteriously vanish (or unwanted pages show up).

How matching actually works

Two rules people get wrong:

  • Longest match wins. For Disallow: /folder/ and Allow: /folder/page, the URL /folder/page is allowed because the Allow rule is longer/more specific.
  • Wildcards are supported. * matches any sequence and $ anchors the end. Disallow: /*.pdf$ blocks every URL ending in .pdf.

Only the first matching User-agent group applies to a given bot, and a bot picks the most specific user-agent group, not *, if one names it.

Common mistakes to catch

  • A trailing-slash rule (Disallow: /blog/) that fails to block /blog without the slash.
  • Relying on Disallow to keep a URL out of search — it only stops crawling.
  • Rules under User-agent: * being silently ignored because a more specific User-agent: Googlebot group exists and overrides them.
  • Case mismatches — paths are case-sensitive.

Example

Given:

User-agent: *
Disallow: /private/
Allow: /private/whitelisted.html

Testing /private/whitelisted.html → Allowed (specific Allow overrides). Testing /private/secret.html → Blocked.

How to use the Robots.txt Tester

  1. 01Paste your robots.txt contents.
  2. 02Enter the URL or path you want to test.
  3. 03Enter the crawler's user-agent (e.g. Googlebot).
  4. 04See whether it is allowed or blocked and which rule matched.

Frequently asked questions

How does the tester decide allowed vs blocked?

It follows the standard rules: the most specific user-agent group applies, then the longest matching path rule wins, and on ties an Allow beats a Disallow. Wildcards (*) and end-anchors ($) are supported.

Does it fetch my live robots.txt?

No — you paste the contents, so nothing leaves your browser. That also lets you test changes before you publish them.