robots.txt Generator
Generate a valid robots.txt with allow/disallow rules, per-user-agent groups, sitemap and crawl-delay. Free, deterministic, in your browser.
Generate robots.txt
Configure crawler access. We build a valid robots.txt.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?
Sitemap: https://example.com/sitemap.xml
What this robots.txt generator does
Common use cases
- 1Controlling crawler access
- 2Blocking admin or cart paths
- 3Declaring your sitemap
- 4Blocking all bots on staging
- 5Setting a crawl-delay
- 6Creating a first robots.txt
Related tools
About robots.txt
robots.txt sits at your site root and tells crawlers which paths they may fetch, grouped by user-agent, and where your sitemap lives. It is the first file most crawlers request.
An important nuance: it controls crawling, not indexing. Disallowing a URL stops well-behaved bots fetching it, but the page can still appear in search if other sites link to it — use a noindex meta tag for that. After publishing, confirm your rules with the robots.txt Tester.
More Free Tools Where This Came From
This utility is one of dozens of free, no-login tools for DNS, email, SEO and developers — all instant and private.
HostCloud.in · Pune, India · Serving 34,987+ Websites Since 2020
Got Questions? We Have Answers.
What can the generator do?
Build an allow-all (with exclusions) or block-all robots.txt, set per-user-agent rules, add Allow/Disallow paths, declare your sitemap and set a crawl-delay.
Does disallow keep a page out of Google?
No. robots.txt controls crawling, not indexing — a disallowed page can still be indexed if linked elsewhere. Use a noindex meta tag to keep it out of search.
Where does the file go?
At your site root (example.com/robots.txt). Confirm the rules with the robots.txt Tester.
Does it use AI or a server?
No. It runs entirely in your browser.