Robots.txt Generator API

Generate a robots.txt (default rules, per-crawler groups, AI crawler blocking, sitemaps), or test whether a crawler may fetch a URL under a robots.txt.

POST /api/v1/tools/robots-txt-generatorMCP robots_txt_generatorWeb & SEO

Create a robots.txt file with per-crawler rules, sitemaps and one-click blocking of AI crawlers like GPTBot and CCBot, then test any URL against the rules.

Input

Send these as a JSON object. Unknown fields are rejected.

mode: generate: build a robots.txt. test: check `url` for `userAgent` against `robotsTxt` (RFC 9309 and Google matching). (string, one of generate, test, default "generate")
defaultPolicy: generate: allow or disallow everything for all crawlers (User-agent: *) before the path rules. (string, one of allow, disallow, default "allow")
disallow: generate: paths to disallow for all crawlers, one per line, each starting with / or *. (string, up to 20000 characters)
allow: generate: paths to allow for all crawlers, one per line. (string, up to 20000 characters)
crawlDelay: generate: Crawl-delay in seconds for all crawlers (ignored by Google). (number, 0 to 86400)
blockAiCrawlers: generate: block all known AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, PerplexityBot, Perplexity-User, Bytespider, meta-externalagent, FacebookBot, Amazonbot, cohere-ai, Diffbot, omgili, Timpibot). (boolean, default false)
groups: generate: extra per-crawler groups as a JSON array, e.g. [{"userAgents":["Googlebot-Image"],"disallow":["/private/"],"allow":[],"crawlDelay":""}]. (string, up to 50000 characters)
sitemaps: generate: absolute sitemap URLs, one per line. (string, up to 20000 characters)
robotsTxt: test: the robots.txt content to test against. (string)
userAgent: test: crawler user-agent or product token, e.g. Googlebot. (string, up to 500 characters)
url: test: the URL or path to check, e.g. https://example.com/admin/login. (string, up to 2048 characters)

Try it in the browser JSON Schema Get an API key

curl https://pithlab.com/api/v1/tools/robots-txt-generator \n  -H "Authorization: Bearer $PITHLAB_KEY" \n  -H "Content-Type: application/json" \n  -d '{"mode":"generate","disallow":"/admin/\n/cart","sitemaps":"https://example.com/sitemap.xml"}'

Example response

{
  "ok": true,
  "result": {
    "text": "User-agent: *\nDisallow: /admin/\nDisallow: /cart\n\nSitemap: https://example.com/sitemap.xml\n",
    "errors": []
  }
}