Claude
Claude can search the web and cites the sources it uses. Anthropic runs three bots: ClaudeBot (training data), Claude-SearchBot (search quality) and Claude-User (fetches pages for users). All honor robots.txt. Anthropic has not named its search provider in product announcements, but TechCrunch reported in March 2025 that Brave Search appeared on Anthropic's subprocessor list.
How Claude sources answers
Anthropic’s web search tool “gives Claude direct access to real-time web content” and “the response includes citations for sources drawn from search results.”[1] Anthropic does not document its ranking. TechCrunch reported that Anthropic added “Brave Search” to its subprocessor list when web search launched, that Simon Willison saw identical citations in Claude and Brave for at least one search, and that Claude’s search function had a parameter named “BraveSearchParams.”[2] Treat Brave as the likely index, not a confirmed one.
Anthropic’s bots
| Bot | What Anthropic says it does | Effect of blocking |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Future content excluded from training datasets |
| Claude-SearchBot | Navigates the web to improve search result quality | Not indexed for search optimization; “may reduce your site’s visibility” |
| Claude-User | Accesses websites when individuals ask Claude questions | Claude cannot retrieve your content for user queries |
Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay directive, and that blocking by IP address “may not work correctly or persistently guarantee an opt-out” because it stops Anthropic reading robots.txt.[3] Vendor-documented The help page names the tokens but does not publish full user-agent strings; match on the token in your logs.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
This example opts out of training while staying eligible for Claude’s search and user fetches. Build your own with the robots.txt builder.
How to be cited by Claude
- Allow Claude-SearchBot and Claude-User. Vendor-documented
- Check how your key pages rank on Brave Search, given the reported partnership. Practitioner practice
- Write self-contained answers with explicit facts, which suit any retrieval system (see content structure). Practitioner practice
Measurement
Anthropic offers no publisher reporting. Track referrals from claude.ai in analytics and include Claude in scheduled prompt tests (see measurement).
Claude's web search tool: what the API documents
Developers can give Claude a server-side web search tool. Anthropic's documentation describes how it behaves, and several details affect which of your pages can be retrieved and how they are cited.[1]
- Citations are always on. Each search result carries a URL and title, and cited text is limited to 150 characters per citation.[1]
- Page age is returned. Results include a
page_agefield, so a clear, accurate date on the page can matter.[1] - Developers and admins can restrict domains. The tool accepts
allowed_domainsorblocked_domains(not both in one request), and organization admins can disable web search or restrict domains in the Console. A site can be absent from a given Claude product because its operator excluded it, not because of anything on the site.[1] - Searches are limited per request. A
max_usesparameter caps the number of searches, and exceeding it returns amax_uses_exceedederror, which bounds how many pages Claude can look at for one question.[1] - Localization is optional. A
user_locationparameter can be supplied to localize results.[1]
The documentation lists three tool versions: web_search_20250305 (basic), web_search_20260209 (dynamic filtering) and web_search_20260318 (adds response_inclusion).[1] Vendor-documented
What follows for site owners
These implications are inferences from the documented behavior, not Anthropic recommendations:
- Keep the sentence that answers a question compact, because cited text is capped at 150 characters.
- Show an accurate publication and modified date, and keep it consistent with structured data (see freshness), because
page_ageis surfaced. - Allow
Claude-SearchBotandClaude-Userin robots.txt if you want to be retrievable, and see AI crawler user agents for what Anthropic does and does not publish about them.[3]
Practitioner practice
Frequently asked questions
What search engine does Claude use?
Anthropic has not said so in product announcements. TechCrunch reported in March 2025 that Brave Search was on Anthropic's subprocessor list and that Claude's citations matched Brave's in testing.
Does blocking ClaudeBot stop Claude from citing me?
No. ClaudeBot is for training. Claude-SearchBot and Claude-User handle search and user fetches and are controlled separately.
Does Anthropic respect Crawl-delay?
Yes, Anthropic says it supports the non-standard Crawl-delay extension.
Can a site be excluded from Claude's web search by the developer using it?
Yes. Anthropic's documentation says developers can set allowed or blocked domains on the web search tool, and organization admins can restrict domains or disable web search in the Console.
See also
References
Pages accessed September 28, 2026 unless a date is given. Sources added on October 31, 2026 were accessed that day. See all sources and our editorial policy.
- ^ "Web search tool". Claude Platform documentation.
- ^ "Anthropic appears to be using Brave to power web search for its Claude chatbot". TechCrunch. Published March 21, 2025.
- ^ "Does Anthropic crawl data from the web, and how can site owners block the crawler?". Anthropic (Claude Help Center).