Free robots.txt Generator — Build It in Seconds
A robots.txt file tells search engines which parts of your site they may crawl. One wrong slash can accidentally block your entire site — or leave admin pages wide open. Add your rules row by row, pick a preset for common setups, and get a clean, validated robots.txt to upload to your site root. Everything runs in your browser.
robots.txt Generator
Build crawler rules in seconds — allow/disallow paths, crawl-delay & sitemap.
100% Free Forever
No Sign-up
No Credit Card
Private — files never leave your browser
Unlimited Use
1 Crawler rules
Each row: the crawler (use * for all), allow or disallow, and the path. Example: Disallow /wp-admin/ blocks the WordPress admin area.
2 Options & presets
Tells crawlers exactly where your sitemap lives.
Seconds between crawler requests. Use 5–10 only if bots overload your server.
Include sitemap line
3 Generate & download
Working…
✅ Your robots.txt
How it works
- Add crawler rules — each row is a user-agent (use
*for all crawlers), an Allow/Disallow action and a path. Use the presets for blogs, stores or a fully open site. - Set options — link your sitemap URL and, only if needed, a crawl-delay to slow aggressive bots.
- Generate & download — the tool assembles a valid robots.txt, checks for common mistakes (like accidentally blocking your whole site), and lets you copy or download it. Upload it to your site’s root:
yourdomain.com/robots.txt.
Everything runs in your browser — your text never leaves this page.
Everyday uses
- Bloggers — block admin and login pages while keeping posts fully crawlable.
- Online stores — keep cart, checkout and account pages out of search results.
- Developers — ship every new site with a correct robots.txt instead of an empty one.
- SEOs — audit client robots.txt files for the classic “Disallow: /” disaster before it tanks traffic.
- Site owners — point crawlers at your sitemap so new pages get discovered faster.
Rule of thumb: remember robots.txt is a polite request, not security — crawlers obey it voluntarily. Never rely on it to hide truly sensitive content; use passwords or noindex for that.