AI robots.txt builder
From AEO.wiki, the Answer Engine Optimization reference · Last reviewed · By Simon White · Published by Local Blitz · How we research
Tick the AI crawlers you want to allow and copy the generated robots.txt. Only documented user-agent tokens are included. The “search visible, no training” preset keeps AI search and user bots allowed while disallowing training crawlers. Everything runs in your browser.
Before you publish
- Googlebot and Bingbot are not listed because blocking them removes you from Google or Bing Search, including AI Overviews and Copilot.[1][2]
- User-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt.[3][4]
- Changes take about 24 hours for OpenAI and Amazon.[3][5]
- Check your CDN or firewall too.[6]
Descriptions of each agent: AI crawlers and robots.txt.
Frequently asked questions
Will blocking training bots hurt my AI visibility?
Search bots are separate, so blocking GPTBot or Google-Extended does not remove you from ChatGPT search or AI Overviews.
Why aren't Bard or Claude-Web in the list?
They don't appear in current vendor documentation, so rules for them do nothing.
See also
References
Pages accessed September 28, 2026 unless a date is given. See all sources and our editorial policy.
- ^ "Google’s common crawlers (including Google-Extended)". Google Search Central. Last updated July 14, 2026.
- ^ "Bing Webmaster Guidelines". Microsoft Bing Webmaster Tools.
- ^ "Overview of OpenAI Crawlers". OpenAI Platform documentation.
- ^ "Perplexity Crawlers". Perplexity documentation.
- ^ "About Amazonbot (Amazonbot, Amzn-SearchBot, Amzn-User)". Amazon Developer.
- ^ "Content Independence Day: no AI crawl without compensation!". Cloudflare Blog. Published July 1, 2025.
Categories: AEO reference and tools