AI robots.txt builder
From AEO.wiki, the Answer Engine Optimization reference · Published · Updated · By the AEO.wiki editorial team · How we research
Tick the AI crawlers you want to allow and copy the generated robots.txt. Only documented user-agent tokens are included. The “search visible, no training” preset keeps AI search and user bots allowed while disallowing training crawlers. Everything runs in your browser.
Before you publish
- Googlebot and Bingbot are not listed because blocking them removes you from Google or Bing Search, including AI Overviews and Copilot.[1][2]
- User-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt.[3][4]
- Changes take about 24 hours for OpenAI and Amazon.[3][5]
- Check your CDN or firewall too.[6]
Descriptions of each agent: AI crawlers and robots.txt.
How crawlers read your robots.txt
- Location and scope. The file must sit at the root of the host (
https://example.com/robots.txt) and applies only to that protocol, host and port: it does not coverhttps://m.example.com/orhttp://example.com/. It must be UTF-8 text.[7] Vendor-documented - Groups. Each group starts with a
User-agentline. A crawler follows the most specific group that matches it; several groups for the same agent are combined.[7] Matching of the product token is case-insensitive, and if no group matches and there is no*group, no rules apply.[8] - Paths. Rules are case-sensitive:
Disallow: /file.aspblocks/file.aspbut not/FILE.asp.[7] When several rules match, the most specific (longest) one wins, and an equivalent allow beats a disallow.[8] - Errors. Under RFC 9309, a 4xx response for robots.txt means the crawler may access everything, while a 5xx or unreachable file means the crawler must assume complete disallow. Crawlers should not use a cached copy for more than 24 hours, and must parse at least 500 KiB.[8] Vendor-documented
- Not security. The protocol “is not a form of access authorization,” and listing a path in robots.txt makes it publicly discoverable.[8]
Common failures the builder cannot catch
| Symptom | Cause |
|---|---|
| A blocked page still appears in Google | robots.txt blocks crawling, not indexing. Google reports “Indexed, though blocked by robots.txt” when others link to the page; to avoid indexing, remove the block and use noindex.[9] |
| A noindex line in robots.txt does nothing | Google does not support a noindex rule in robots.txt; use a meta tag or X-Robots-Tag header on a crawlable page.[10] |
| Rules for ChatGPT-User or Perplexity-User are ignored | These fetchers act on a user’s request: OpenAI says robots.txt rules may not apply to ChatGPT-User,[3] and Perplexity says Perplexity-User generally ignores them.[4] |
| Anthropic bots still arrive after an IP block | Anthropic says blocking IP addresses may not work and can stop its bots from reading robots.txt; the documented opt-out is a robots.txt rule.[11] |
| The site vanishes from crawlers after an outage | A 5xx on /robots.txt makes compliant crawlers assume complete disallow.[8] |
| Rules work on www but not the apex domain | Each host has its own file.[7] |
Frequently asked questions
Will blocking training bots hurt my AI visibility?
Search bots are separate, so blocking GPTBot or Google-Extended does not remove you from ChatGPT search or AI Overviews.
Why aren't Bard or Claude-Web in the list?
They don't appear in current vendor documentation, so rules for them do nothing.
See also
References
Pages accessed September 28, 2026 unless a date is given. See all sources and our editorial policy.
- ^ "Google’s common crawlers (including Google-Extended)". Google Search Central. Last updated July 14, 2026.
- ^ "Bing Webmaster Guidelines". Microsoft Bing Webmaster Tools.
- ^ "Overview of OpenAI Crawlers". OpenAI Platform documentation.
- ^ "Perplexity Crawlers". Perplexity documentation.
- ^ "About Amazonbot (Amazonbot, Amzn-SearchBot, Amzn-User)". Amazon Developer.
- ^ "Content Independence Day: no AI crawl without compensation!". Cloudflare Blog. Published July 1, 2025.
- ^ "Create and submit a robots.txt file". Google Crawling Infrastructure. Last updated November 21, 2025. Accessed October 29, 2026.
- ^ "RFC 9309: Robots Exclusion Protocol". IETF (RFC Editor). September 2022. Accessed October 29, 2026.
- ^ "Page indexing report". Search Console Help. Accessed October 29, 2026.
- ^ "Block Search indexing with noindex". Google Search Central. Last updated December 10, 2025. Accessed October 29, 2026.
- ^ "Does Anthropic crawl data from the web, and how can site owners block the crawler?". Anthropic (Claude Help Center). Accessed October 29, 2026.
Categories: AEO reference and tools