Loona Automation logoLoona Developer Tools
Home › Developer Tools › Robots.txt Generator

Robots.txt Generator - Crawler Rules Without the Syntax Traps

In short: the Robots.txt Generator builds a valid robots.txt from presets and per-agent rules - allow and disallow paths per crawler, plus your sitemap line - and runs sanity checks on the result before you ship it.

One wrong line in robots.txt can hide a whole site from Google - or fail to hide anything at all. Build it here, with presets for the common cases and checks on the output.

Private by design: everything in this tool happens 100% in your browser with JavaScript. Your input is never uploaded to a server, never stored and never seen by us - close the tab and everything is gone.

How to use the Robots.txt Generator

  1. Pick a preset, or start from Allow everything and customize.
  2. Add a group for any specific crawler (for example GPTBot) with its own rules.
  3. List disallow and allow paths, one per line - paths start with /.
  4. Add your sitemap URL and click Generate.
  5. Run through the sanity-check notes, then place the file at yourdomain.com/robots.txt.

How it works

A robots.txt file is a set of groups: each group names one or more user-agents (crawlers) followed by Disallow and Allow path rules that apply to them. An empty Disallow means everything is allowed; Disallow: / blocks the whole site for that agent. Rules are matched by prefix, and Google also supports Allow to carve exceptions out of a broader Disallow. This generator emits groups in the standard format, appends the Sitemap line, and then sanity-checks its own output: unknown fields, paths that do not start with /, and sitemap lines that are not full URLs are all flagged. Remember robots.txt is a request, not a lock - well-behaved crawlers obey it, malicious ones may not, so never rely on it to protect secrets.

Frequently Asked Questions

Is this generator free?

Yes - completely free, no sign-up.

Where does robots.txt go?

At the root of your domain: https://yourdomain.com/robots.txt - nowhere else is read.

Does robots.txt block pages from search results completely?

It blocks crawling, not indexing: a blocked URL can still appear in results (without a description) if other sites link to it. Use a noindex tag to keep a page out entirely.

Can I block AI crawlers?

The AI preset adds groups disallowing GPTBot, CCBot and Google-Extended. Compliant crawlers honor it; it cannot stop scrapers that ignore robots.txt.

Is anything uploaded?

No. Generation and checks run in your browser.

Related Developer Tools

Embed this tool on your website — free

Add this tool to your own website or blog — it is completely free, needs no sign-up, and always shows the latest version. Copy the code below and paste it into your page where you want the tool to appear. Keeping the small “Powered by Loona Automation” credit line under the embed helps keep this tool free — but you are free to style or remove it; the tool works either way.