robots.txt Tester
Fetch a site's robots.txt and test whether any path is crawlable for a given user-agent, using the correct longest-match rule. Free, live.
Test robots.txt
Enter a site. We fetch its robots.txt and test whether a path is crawlable.
What this robots.txt tester does
Common use cases
- 1Checking a page is crawlable
- 2Debugging pages missing from search
- 3Testing Disallow rules before deploying
- 4Confirming a sitemap is declared
- 5Auditing bot access per user-agent
- 6Verifying a robots.txt change
Related tools
About robots.txt
robots.txt lives at the root of a site and tells crawlers which paths they may or may not fetch, grouped by user-agent. It is a crawling directive, not a security control — and importantly, disallowing a page does not guarantee it stays out of the index if other sites link to it (use a noindex tag for that).
Crawlers apply the most specific (longest) matching rule, with Allow able to override a broader Disallow. This tester fetches the live file, parses its groups and sitemaps, and evaluates any path against those rules for the user-agent you choose, so you can confirm exactly what a crawler is permitted to fetch.
More Free Tools Where This Came From
This utility is one of dozens of free, no-login tools for DNS, email, SEO and developers — all instant and private.
HostCloud.in · Pune, India · Serving 34,987+ Websites Since 2020
Got Questions? We Have Answers.
What does the robots.txt tester do?
It fetches the live robots.txt, parses its groups and sitemaps, and evaluates whether a specific path is allowed or blocked for the user-agent you choose.
How are conflicting rules resolved?
Crawlers apply the most specific (longest) matching rule, and an Allow can override a broader Disallow. The tester uses the same logic, with * and $ wildcards.
Does disallow keep a page out of Google?
Not necessarily. robots.txt controls crawling, not indexing — a disallowed page can still be indexed if linked elsewhere. Use a noindex tag to keep it out of search.
Is it live?
Yes. The robots.txt is fetched server-side at the moment you test.