Robots.txt Tester (100% FREE)
This robots.txt tester checks whether your rules let a crawler request a URL, and shows which rule applies and why.
Paste your robots.txt, choose a crawler, and test a URL path.
Free, no signup. Runs in your browser, nothing is sent anywhere. It checks the rules you paste, not a live crawl.
Paste your robots.txt and a URL to see whether that crawler may request it.
How it works
How to use the Robots.txt Tester in three steps.
- Step 1
Paste your rules
Copy the text from your site's /robots.txt file.
- Step 2
Select a crawler and URL
Try Googlebot or the exact user agent you want to check.
- Step 3
Read the matched rule
The tester shows whether the rules allow that path, not whether the page will be indexed or rank.
Examples
How the rules actually resolve.
Rules: User-agent: *, Disallow: /drafts/, Allow: /drafts/public/, Disallow: /*.pdf$
A more specific Allow
Input: Googlebot, /drafts/public/guide The longer, more specific rule wins over Disallow: /drafts/.
ResultAllowed by Allow: /drafts/public/
Still blocked
Input: Googlebot, /drafts/secret/ Only the Disallow rule matches this path.
ResultDisallowed by Disallow: /drafts/
The end anchor
Input: Googlebot, /files/report.pdf?v=2 Disallow: /*.pdf$ only matches URLs that end in .pdf, and this one ends in a query string.
ResultAllowed
Good to know
Robots.txt, explained plainly.
What robots.txt does
It tells compliant crawlers which paths they may request. It lives at the root of a site, like https://example.com/robots.txt, and applies to that scheme, host and port. It controls crawling, not access: it's not a password and won't reliably hide files. A blocked URL can still appear in search results without its content.
How Allow and Disallow are evaluated
A crawler finds the user agent group that matches it most specifically, or the * group, then compares the path with that group's rules. Under the Robots Exclusion Protocol (RFC 9309) the most specific matching rule wins, and Allow wins a tie. Wildcards (*) and the end marker ($) change what matches. A test that just looks for the first Disallow line is wrong.
Robots.txt, noindex and sitemaps are different
Robots.txt controls crawl requests. A noindex directive asks search engines not to index a page, but the crawler has to be allowed in to see it. A sitemap lists URLs you want discovered; it doesn't override a Disallow. To protect private content, require a login.
Methodology
The tester follows RFC 9309 and Google's robots.txt documentation: case insensitive user agent matching, merged groups, case sensitive paths, * and $, longest match with Allow winning ties, and /robots.txt always allowed.
FAQ
Questions, answered.
How do I test a URL in robots.txt?
Paste your site's rules, pick the crawler and enter the path. The tester applies the matching rules and shows the line responsible for the result.
Does Disallow: / block my entire site?
For the matching user agent group, it blocks crawling of every path unless a more specific Allow rule applies. Test a few paths before you deploy it.
Can I block Google from indexing with robots.txt?
Not reliably. A blocked URL can still be indexed from links elsewhere, without its content. Use a noindex directive or put the page behind a login, depending on your goal.
What is the difference between robots.txt and a meta robots tag?
Robots.txt is a site level file about crawling. A meta robots tag sits in a page and tells search engines about indexing, once they're allowed to crawl it.
Should I add my sitemap to robots.txt?
You can add a Sitemap: line with the full URL. It helps discovery but doesn't override Disallow rules.
Does a robots.txt tester show if Google has indexed my page?
No. It checks permissions. Use Google Search Console's URL inspection for indexing status.
Why is Googlebot allowed but another bot blocked?
Different user agent groups can have different rules. The tester uses the group that matches the crawler you pick.
What if robots.txt is missing?
Under the standard, a missing file means no crawl restrictions. Some fetch errors are treated differently by crawlers, so don't assume every error means allowed.
Is robots.txt a way to protect private files?
No. The file is public and crawlers don't enforce access. Use authentication and server permissions.
Technical SEO giving you trouble?
I fix crawling, indexing and site structure problems as part of ongoing SEO.
See SEO services