Robots.txt Tester - Allowed or Blocked, and Exactly Why
A robots.txt file is easy to write and hard to reason about. This tester answers the only question that matters for a given crawler and URL.
How to use the Robots.txt Tester
- Paste your robots.txt content.
- Enter the full URL to test and the user-agent name (Googlebot, GPTBot, *).
- Click Test this URL.
- Read the verdict and the matching rule that produced it.
How it works
Matching follows the standard (RFC 9309, as Google implements it): first the crawler's group is chosen - the group naming the most specific matching user-agent token wins over the * group - then, within that group, the rule whose path pattern matches the URL's path and is longest wins; on a tie, Allow beats Disallow. Patterns support * as a wildcard and a trailing $ as an end anchor. An empty Disallow matches nothing, which is why 'Disallow:' alone allows everything.
Frequently Asked Questions
Is this tester free?
Yes - completely free, no sign-up.
Which matching rules does it follow?
RFC 9309 / Google's implementation: most specific agent group, longest matching rule, Allow wins ties, * and $ pattern support.
My URL is blocked here but indexed in Google - why?
Robots blocks crawling, not indexing. A blocked page can still be indexed from links, shown without a description.
Does it fetch my live robots.txt?
No - paste the content. Browsers cannot fetch arbitrary sites' files (CORS), and pasting also lets you test a draft before publishing it.
Is anything uploaded?
No. Testing runs in your browser.