Robots.txt Generator
Build a robots.txt file and test which URLs crawlers may fetch.
All crawlers (User-agent: *)
Rules for specific crawlers
Block AI crawlers
Test a URL
Use via API
Generate a robots.txt (default rules, per-crawler groups, AI crawler blocking, sitemaps), or test whether a crawler may fetch a URL under a robots.txt. Send JSON, get JSON back.
POST https://pithlab.com/api/v1/tools/robots-txt-generator
curl https://pithlab.com/api/v1/tools/robots-txt-generator \
-H "Authorization: Bearer $PITHLAB_KEY" \
-H "Content-Type: application/json" \
-d '{"mode":"generate","disallow":"/admin/\n/cart","sitemaps":"https://example.com/sitemap.xml"}'About the Robots.txt Generator
Create a robots.txt file with per-crawler rules, sitemaps and one-click blocking of AI crawlers like GPTBot and CCBot, then test any URL against the rules.
How to use the Robots.txt Generator
- Choose whether crawlers are allowed or blocked by default, and list any paths to disallow or allow for all crawlers.
- Add a group for each crawler that needs its own rules, such as Bingbot, with its own paths and crawl delay.
- Tick the AI crawlers you want to block, or use Block all AI crawlers.
- Add your sitemap URLs, then copy the file or download it as robots.txt and upload it to the root of your site.
- Use the tester to check whether a crawler may fetch a given URL, and which rule decides it.
Examples
Input
Allow all, disallow /admin/, block GPTBot and CCBot, sitemap https://example.com/sitemap.xml
Output
User-agent: * Disallow: /admin/ # AI crawlers User-agent: GPTBot User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Input
User-agent: * Disallow: /admin Allow: /admin/public Test: Googlebot /admin/public/page
Output
Allowed. Line 3: "Allow: /admin/public" is the longest matching rule.
Frequently asked questions
Does robots.txt stop pages appearing in Google?
Not reliably. robots.txt controls crawling, not indexing. A blocked page can still be listed if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag instead.
How are conflicting Allow and Disallow rules resolved?
Following RFC 9309 and Google, the rule with the longest matching path wins, whatever order the rules are in. If an Allow and a Disallow rule are the same length, Allow wins. * matches any characters and a $ at the end anchors the end of the URL.
Will blocking AI crawlers work?
Well-behaved crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended respect robots.txt. It is a request, not access control, so crawlers that ignore it are not stopped. Blocking Google-Extended does not affect Google Search.
Does Google support Crawl-delay?
No. Google ignores Crawl-delay. Bing, Yandex and some other crawlers honour it as the number of seconds to wait between requests.
Where does the file go?
At the root of each host, for example https://example.com/robots.txt. A subdomain such as shop.example.com needs its own file. The file is built in your browser and nothing is uploaded by this tool.