Loona Automation logoLoona Developer Tools
Home › Developer Tools › Robots.txt Tester

Robots.txt Tester - Allowed or Blocked, and Exactly Why

In short: the Robots.txt Tester applies the real matching rules - the most specific user-agent group, then the longest matching path rule, with Allow winning ties - and shows the verdict plus the exact rule that decided it.

A robots.txt file is easy to write and hard to reason about. This tester answers the only question that matters for a given crawler and URL.

Private by design: everything in this tool happens 100% in your browser with JavaScript. Your input is never uploaded to a server, never stored and never seen by us - close the tab and everything is gone.

How to use the Robots.txt Tester

  1. Paste your robots.txt content.
  2. Enter the full URL to test and the user-agent name (Googlebot, GPTBot, *).
  3. Click Test this URL.
  4. Read the verdict and the matching rule that produced it.

How it works

Matching follows the standard (RFC 9309, as Google implements it): first the crawler's group is chosen - the group naming the most specific matching user-agent token wins over the * group - then, within that group, the rule whose path pattern matches the URL's path and is longest wins; on a tie, Allow beats Disallow. Patterns support * as a wildcard and a trailing $ as an end anchor. An empty Disallow matches nothing, which is why 'Disallow:' alone allows everything.

Frequently Asked Questions

Is this tester free?

Yes - completely free, no sign-up.

Which matching rules does it follow?

RFC 9309 / Google's implementation: most specific agent group, longest matching rule, Allow wins ties, * and $ pattern support.

My URL is blocked here but indexed in Google - why?

Robots blocks crawling, not indexing. A blocked page can still be indexed from links, shown without a description.

Does it fetch my live robots.txt?

No - paste the content. Browsers cannot fetch arbitrary sites' files (CORS), and pasting also lets you test a draft before publishing it.

Is anything uploaded?

No. Testing runs in your browser.

Related Developer Tools

Embed this tool on your website — free

Add this tool to your own website or blog — it is completely free, needs no sign-up, and always shows the latest version. Copy the code below and paste it into your page where you want the tool to appear. Keeping the small “Powered by Loona Automation” credit line under the embed helps keep this tool free — but you are free to style or remove it; the tool works either way.