Robots.txt Tester: Is a URL Allowed for Googlebot?
Test a URL against robots.txt rules for any crawler.
Use via API
Test whether a URL path is allowed for a crawler under a site's robots.txt (fetched, or passed in), using Google's longest-match rules. Send JSON, get JSON back; this one reaches the network from PithLab’s server.
POST https://pithlab.com/api/v1/tools/robots-txt-tester
curl https://pithlab.com/api/v1/tools/robots-txt-tester \
-H "Authorization: Bearer $PITHLAB_KEY" \
-H "Content-Type: application/json" \
-d '{"robotsTxt":"User-agent: *\nDisallow: /admin/\nAllow: /admin/help","path":"/admin/help","userAgent":"Googlebot"}'About the Robots.txt Tester
Fetch a site's robots.txt or paste your own, then test whether a URL is allowed for Googlebot, Bingbot, GPTBot or ClaudeBot using Google's longest-match rules.
How to use the Robots.txt Tester
- Enter a site, such as example.com, and press Fetch, or switch to Paste and paste a robots.txt file (pasted files are tested in your browser).
- Choose a crawler, such as Googlebot or GPTBot, and enter the URL or path to test.
- See whether it's allowed, which group and rule decided it, every group in the file, the sitemaps it lists and any lines crawlers will ignore.
Examples
Input
/admin/help for Googlebot
Output
Allowed. Line 3, "Allow: /admin/help", is the longest matching rule in the group (User-agent: *).
Input
/blog/post for GPTBot
Output
Blocked. Line 6, "Disallow: /", is the longest matching rule in the group (User-agent: GPTBot).
Frequently asked questions
How does Google decide between Allow and Disallow?
Google uses the most specific group for the crawler (falling back to User-agent: *), then the matching rule with the longest path wins. If an Allow and a Disallow are equally long, Allow wins. * matches any characters and $ marks the end of the URL.
Does Disallow stop a page appearing in Google?
Not always. Disallow stops crawling, but a blocked URL can still be indexed from links elsewhere, without a description. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead.
How do I block AI crawlers like GPTBot and ClaudeBot?
Add a group for each one, such as User-agent: GPTBot followed by Disallow: /. Test it here by choosing that crawler. Google-Extended controls whether Google may use your content for Gemini training.
What if the site has no robots.txt?
A 404 means crawlers may fetch everything. A server error (5xx) is different: Google treats the whole site as blocked until robots.txt can be read again.