AI crawler user agents
This page is a lookup for what each vendor publishes about its crawlers: the robots.txt token, a user-agent string where one is published, how to verify a request and whether the crawler follows robots.txt. For rules and blocking decisions, see AI crawlers and robots.txt.
OpenAI
| Token | Purpose | User-agent string (as documented) | IP list |
|---|---|---|---|
OAI-SearchBot | Surfaces sites in ChatGPT search results | OAI-SearchBot/1.4; +https://openai.com/searchbot | openai.com/searchbot.json |
GPTBot | Collects content that may be used to train generative AI models | GPTBot/1.4; +https://openai.com/gptbot | openai.com/gptbot.json |
ChatGPT-User | Fetches pages when a ChatGPT user (or a GPT Action) asks for them; not used for automatic crawling or for Search opt-outs | ChatGPT-User/1.0; +https://openai.com/bot | openai.com/chatgpt-user.json |
OAI-AdsBot | Validates the safety of pages submitted as ads on ChatGPT; not used to train models | OAI-AdsBot/1.0; +https://openai.com/adsbot | openai.com/adsbot.json |
OpenAI states that robots.txt changes take about 24 hours to take effect, that a robots.txt fetch adds a robots.txt marker to the user-agent, and that because ChatGPT-User requests are initiated by users, robots.txt rules may not apply to it.[1] Vendor-documented OpenAI's full strings take the form Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, so match on the token, not the whole string. Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers but can still appear as navigational links.
Perplexity
Perplexity documents two user agents. PerplexityBot (PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) is designed to surface and link websites in Perplexity search results and is not used to train foundation models. Perplexity-User (Perplexity-User/1.0) supports user actions inside Perplexity and, because a user requested the fetch, generally ignores robots.txt rules. Each has a published IP list (perplexitybot.json and perplexity-user.json). Changes to robots.txt can take up to 24 hours to apply.[2] Perplexity recommends combining user-agent and IP checks in a firewall rule.[2] Vendor-documented
Google lists its crawlers in one reference.[3] The tokens most relevant to AI work:
Googlebot: Search indexing. This is the crawler behind AI Overviews and AI Mode eligibility, since pages must be indexed and snippet-eligible.[4]Google-Extended: a robots.txt token only, with no separate user-agent string. It controls whether content Googlebot fetched can be used for Gemini training and for grounding in Gemini Apps and the Vertex AI API for Gemini. It does not affect inclusion or ranking in Google Search.[3]GoogleOther(and the Image and Video variants),Google-InspectionToolandGoogle-CloudVertexBot, each with its own purpose in Google's reference.[3]
Google publishes crawler IP ranges in common-crawlers.json and says reverse DNS for its crawlers resolves to names matching crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.[3] Google also says it intends to support the AIPREF signal, written in robots.txt as user-agent: __usage_ai-training, from early 2027.[3] Vendor-documented
Microsoft Bing
Bing's crawler reference lists Bingbot, AdIdxBot (ads), BingPreview, MicrosoftPreview and BingVideoPreview, each with a documented user-agent string. Microsoft provides a Verify Bingbot tool and a crawl control setting in Bing Webmaster Tools.[5] See Copilot and Bing for how Bing relates to Copilot. Vendor-documented
Apple
Applebot supports Spotlight, Siri and Safari. Applebot-Extended is a token that never crawls: disallowing it opts content out of use for training Apple's foundation models. Apple says reverse DNS for Applebot resolves under *.applebot.apple.com, that Applebot follows Googlebot rules if robots.txt does not mention Applebot, and that it ignores crawl-delay.[6] Vendor-documented See voice assistants for the Siri controls.
Anthropic
Anthropic's help center names three user agents, ClaudeBot, Claude-User and Claude-SearchBot, and says each can be controlled in robots.txt. It says Anthropic's bots honor robots.txt directives, support the non-standard Crawl-delay extension and do not try to bypass CAPTCHAs. The page does not list full user-agent strings or IP ranges.[7] Vendor-documented See Claude.
Common Crawl
Common Crawl's crawler identifies as CCBot/2.0 (https://commoncrawl.org/faq/), honors a CCBot robots.txt token, publishes an IP list at index.commoncrawl.org/ccbot.json and resolves to *.crawl.commoncrawl.org in reverse DNS.[8] Vendor-documented
Matching and verifying in logs
A user-agent string is self-reported and can be copied by anyone. Every vendor above that publishes IP lists or reverse DNS masks does so for this reason, and Perplexity explicitly recommends combining the two checks.[2] Two points follow from the vendor pages:
Google-ExtendedandApplebot-Extendedare control tokens only, so neither will ever appear as a user agent in your logs.[3][6]- User-initiated fetchers (
ChatGPT-User,Perplexity-User) are documented as not necessarily following robots.txt, so robots.txt is not a reliable way to stop them.[1][2]
A starting log filter built from the tokens above (practice, not vendor guidance): OAI-SearchBot|GPTBot|ChatGPT-User|OAI-AdsBot|PerplexityBot|Perplexity-User|ClaudeBot|Claude-User|Claude-SearchBot|Applebot|bingbot|Googlebot|CCBot. Count matches, then verify a sample of each against the vendor's IP list before drawing conclusions. Practitioner practice
Frequently asked questions
Which AI crawler should I allow to appear in ChatGPT search?
OpenAI documents OAI-SearchBot as the crawler that surfaces sites in ChatGPT search results. GPTBot is a separate crawler for training data, so the two can be set independently in robots.txt.
Why do I never see Google-Extended in my logs?
Google-Extended is a robots.txt token only. Google fetches pages with its usual crawlers and then applies the token to decide whether the content can be used for Gemini training and grounding.
Can I block ChatGPT-User or Perplexity-User with robots.txt?
Both vendors document these as fetchers started by a person's request, and say robots.txt rules may not apply or are generally ignored. Use firewall rules with verified IP ranges if you need to block them.
See also
References
Pages accessed October 31, 2026. See all sources and our editorial policy.
- ^ "Overview of OpenAI Crawlers". OpenAI Platform documentation.
- ^ "Perplexity Crawlers". Perplexity documentation.
- ^ "Google’s common crawlers (including Google-Extended)". Google Search Central.
- ^ "Optimizing your website for generative AI features on Google Search". Google Search Central. Added May 15, 2026; last updated July 10, 2026.
- ^ "Which Crawlers Does Bing Use?". Microsoft Bing Webmaster Tools.
- ^ "About Applebot". Apple Support. Page dated September 4, 2026.
- ^ "Does Anthropic crawl data from the web, and how can site owners block the crawler?". Anthropic (Claude Help Center).
- ^ "CCBot". Common Crawl.