Robots.txt Generator
Build a correct robots.txt in a minute — keep private paths out of search, choose which AI and SEO crawlers may visit, add your sitemap — then check any robots.txt and test URLs the way Googlebot reads them.
At a glance
- Generates robots.txt from simple choices: default access, paths to block with exceptions, an optional Crawl-delay and your sitemap URLs.
- Lets you allow or block 26 named crawlers in four groups — search engines, AI search assistants (OAI-SearchBot, Claude-SearchBot, PerplexityBot), AI training bots (GPTBot, ClaudeBot, Google-Extended, CCBot …) and SEO crawlers (AhrefsBot, SemrushBot …).
- Presets for WordPress, online shops, staging sites, and blocking AI training while staying in search and AI answers.
- Validates any robots.txt with line numbers: rules outside a group, unsupported noindex, misspellings, relative sitemaps, blocked CSS and JavaScript, and Google’s 500 KiB limit.
- Tests a URL against every crawler using Google’s matching rules (RFC 9309): the most specific user-agent group, the longest matching path, Allow winning ties — and shows the line that decided.
- Loads a site’s live robots.txt for checking — directly from your browser when the site allows it, otherwise through Jina Reader. Building and testing happen on your device.
Step by step
How to create a robots.txt file
- 1
Pick a starting point
Choose a preset such as WordPress or Block AI training, or start from Allow everything.
- 2
Add your rules
List paths to keep out of search (like /admin/), allow or block specific crawlers, and add your sitemap URL.
- 3
Test it
Click Test a URL against this file to check important pages are still crawlable.
- 4
Upload
Download robots.txt and upload it to your site’s root so it opens at example.com/robots.txt.
Features
Everything you need, nothing you don’t
Correct by construction
Groups, paths and sitemap lines are written the way Google, Bing and AI crawlers expect, with full sitemap URLs.
AI crawler control
Opt out of AI training without disappearing from ChatGPT, Claude or Perplexity search answers — or block AI completely.
Smart presets
WordPress, online shop, staging and AI presets give a sensible starting point in one click.
Validator with line numbers
Finds the mistakes that silently break robots.txt — and jumps to the line.
URL tester
See which crawlers may fetch a page and which rule decides, before you upload the file.
Check any live site
Load a website’s robots.txt to audit it, or to start editing your own.
robots.txt rules at a glance
| Line | Meaning | Example |
|---|---|---|
| User-agent | Which crawler the following rules are for; * means all | User-agent: GPTBot |
| Disallow | Paths the crawler must not fetch | Disallow: /admin/ |
| Allow | An exception inside a disallowed path | Allow: /admin/help/ |
| * | Matches any characters | Disallow: /*?sort= |
| $ | Marks the end of the URL | Disallow: /*.pdf$ |
| Sitemap | Full URL of a sitemap, anywhere in the file | Sitemap: https://example.com/sitemap.xml |
What robots.txt can and can’t do
robots.txt controls crawling, not indexing. A blocked page can still appear in Google as a bare link if other sites link to it, and Google can’t see a noindex tag on a page it isn’t allowed to fetch. To keep a page out of search results, allow crawling and add <meta name="robots" content="noindex"> instead.
It is also public and voluntary: well-behaved crawlers such as Googlebot, Bingbot, GPTBot and ClaudeBot obey it, but it is not access control. Never list secret paths in it, and protect private areas with a login.
FAQ
Frequently asked questions
Where do I put robots.txt?
How do I block ChatGPT and other AI bots?
Will blocking Google-Extended remove my site from Google Search?
Does Google follow Crawl-delay?
Why does the validator say noindex isn’t supported?
Is blocking CSS and JavaScript a problem?
Keep going
Related tools
Sitemap Generator
Free XML sitemap generator: crawl your site or paste URLs, skip noindex, redirected and robots.txt-blocked pages, add lastmod and images, ready for Google.
Website SEO Checker
Free website SEO checker: audit any page’s title, meta description, headings, canonical, robots.txt, schema, Open Graph and AI crawler access, with fixes.
UTM Builder
Build UTM campaign links for Google Analytics 4: presets for WhatsApp, Instagram, email and ads, the GA4 channel each lands in, and a QR code for it.
SSL Checker
Check any website’s SSL certificate: expiry date and days left, issuer, covered domains, revocation and CAA, plus a live HTTPS test from your browser.
URL Shortener
Free URL shortener: make short links with a custom ending, QR code, password, expiry and click stats — or check where any short link really goes.
Image Compressor
Compress JPG, PNG, WebP and AVIF images online for free. Shrink photos by up to 90% or hit an exact KB target — right in your browser, with no upload.
Last updated Report a problem or suggest a feature