Robots.txt Tester API

Test whether a URL path is allowed for a crawler under a site's robots.txt (fetched, or passed in), using Google's longest-match rules.

POST /api/v1/tools/robots-txt-testerMCP robots_txt_testerWeb & SEOLive network check

Fetch a site's robots.txt or paste your own, then test whether a URL is allowed for Googlebot, Bingbot, GPTBot or ClaudeBot using Google's longest-match rules.

Input

Send these as a JSON object. Unknown fields are rejected.

site: Site whose /robots.txt to fetch, such as example.com. Leave out to test robotsTxt instead. (string, up to 2048 characters)
robotsTxt: robots.txt content to test instead of fetching one. (string, up to 512000 characters)
path: The URL or path to test. (string, up to 2048 characters, default "/")
userAgent: Crawler user-agent token, such as Googlebot, Bingbot, GPTBot, ClaudeBot or *. (string, up to 100 characters, default "Googlebot")

Try it in the browser JSON Schema Get an API key

curl https://pithlab.com/api/v1/tools/robots-txt-tester \n  -H "Authorization: Bearer $PITHLAB_KEY" \n  -H "Content-Type: application/json" \n  -d '{"robotsTxt":"User-agent: *\nDisallow: /admin/\nAllow: /admin/help","path":"/admin/help","userAgent":"Googlebot"}'