Claude
Claude can search the web and cites the sources it uses. Anthropic runs three bots: ClaudeBot (training data), Claude-SearchBot (search quality) and Claude-User (fetches pages for users). All honor robots.txt. Anthropic has not named its search provider in product announcements, but TechCrunch reported in March 2025 that Brave Search appeared on Anthropic's subprocessor list.
How Claude sources answers
Anthropic’s web search tool “gives Claude direct access to real-time web content” and “the response includes citations for sources drawn from search results.”[1] Anthropic does not document its ranking. TechCrunch reported that Anthropic added “Brave Search” to its subprocessor list when web search launched, that Simon Willison saw identical citations in Claude and Brave for at least one search, and that Claude’s search function had a parameter named “BraveSearchParams.”[2] Treat Brave as the likely index, not a confirmed one.
Anthropic’s bots
| Bot | What Anthropic says it does | Effect of blocking |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Future content excluded from training datasets |
| Claude-SearchBot | Navigates the web to improve search result quality | Not indexed for search optimization; “may reduce your site’s visibility” |
| Claude-User | Accesses websites when individuals ask Claude questions | Claude cannot retrieve your content for user queries |
Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay directive, and that blocking by IP address “may not work correctly or persistently guarantee an opt-out” because it stops Anthropic reading robots.txt.[3] Vendor-documented The help page names the tokens but does not publish full user-agent strings; match on the token in your logs.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
This example opts out of training while staying eligible for Claude’s search and user fetches. Build your own with the robots.txt builder.
How to be cited by Claude
- Allow Claude-SearchBot and Claude-User. Vendor-documented
- Check how your key pages rank on Brave Search, given the reported partnership. Practitioner practice
- Write self-contained answers with explicit facts, which suit any retrieval system (see content structure). Practitioner practice
Measurement
Anthropic offers no publisher reporting. Track referrals from claude.ai in analytics and include Claude in scheduled prompt tests (see measurement).
Frequently asked questions
What search engine does Claude use?
Anthropic has not said so in product announcements. TechCrunch reported in March 2025 that Brave Search was on Anthropic's subprocessor list and that Claude's citations matched Brave's in testing.
Does blocking ClaudeBot stop Claude from citing me?
No. ClaudeBot is for training. Claude-SearchBot and Claude-User handle search and user fetches and are controlled separately.
Does Anthropic respect Crawl-delay?
Yes, Anthropic says it supports the non-standard Crawl-delay extension.
See also
References
Pages accessed September 28, 2026 unless a date is given. See all sources and our editorial policy.
- ^ "Web search tool". Claude Platform documentation.
- ^ "Anthropic appears to be using Brave to power web search for its Claude chatbot". TechCrunch. Published March 21, 2025.
- ^ "Does Anthropic crawl data from the web, and how can site owners block the crawler?". Anthropic (Claude Help Center).