Robots.txt Generator - Crawler Rules Without the Syntax Traps
One wrong line in robots.txt can hide a whole site from Google - or fail to hide anything at all. Build it here, with presets for the common cases and checks on the output.
How to use the Robots.txt Generator
- Pick a preset, or start from Allow everything and customize.
- Add a group for any specific crawler (for example GPTBot) with its own rules.
- List disallow and allow paths, one per line - paths start with /.
- Add your sitemap URL and click Generate.
- Run through the sanity-check notes, then place the file at yourdomain.com/robots.txt.
How it works
A robots.txt file is a set of groups: each group names one or more user-agents (crawlers) followed by Disallow and Allow path rules that apply to them. An empty Disallow means everything is allowed; Disallow: / blocks the whole site for that agent. Rules are matched by prefix, and Google also supports Allow to carve exceptions out of a broader Disallow. This generator emits groups in the standard format, appends the Sitemap line, and then sanity-checks its own output: unknown fields, paths that do not start with /, and sitemap lines that are not full URLs are all flagged. Remember robots.txt is a request, not a lock - well-behaved crawlers obey it, malicious ones may not, so never rely on it to protect secrets.
Frequently Asked Questions
Is this generator free?
Yes - completely free, no sign-up.
Where does robots.txt go?
At the root of your domain: https://yourdomain.com/robots.txt - nowhere else is read.
Does robots.txt block pages from search results completely?
It blocks crawling, not indexing: a blocked URL can still appear in results (without a description) if other sites link to it. Use a noindex tag to keep a page out entirely.
Can I block AI crawlers?
The AI preset adds groups disallowing GPTBot, CCBot and Google-Extended. Compliant crawlers honor it; it cannot stop scrapers that ignore robots.txt.
Is anything uploaded?
No. Generation and checks run in your browser.
Related Developer Tools
Meta Tag Generator
Generate a ready-to-paste HTML meta block - title, description, robots, canonical - with a SERP-style preview.
Open Graph & Twitter Card Generator
Build Open Graph and Twitter Card tags with a card preview, so shared links look right everywhere.
Robots.txt Generator
Build a valid robots.txt with per-agent allow and disallow rules and a sitemap line.