Claude-SearchBot is the name an Anthropic crawler uses to identify itself when it fetches web pages. Crawlers are programs that fetch web pages automatically. Claude-SearchBot works for Claude’s web search: the feature Claude uses to answer questions with current web content and to cite the pages it used as sources. Like OAI-SearchBot from OpenAI and PerplexityBot from Perplexity, Claude-SearchBot is an AI index crawler: it captures pages in advance so that an AI system can draw on them for answers later.
Three bots with separate jobs
Anthropic runs three bots, each with its own name, known as a token, and its own purpose. Claude-SearchBot captures pages so that Anthropic’s system can add them to its index for web search, a web index: a database that stores crawled pages in processed form so that matching content can be found quickly.
A different bot, ClaudeBot, collects web content that could contribute to training Anthropic’s AI models. According to Anthropic, blocking ClaudeBot signals that a site’s future content should be excluded from its training datasets; a rule for ClaudeBot leaves Claude-SearchBot unaffected, and vice versa. Claude-User may visit websites when someone asks Claude a question; it is a user-triggered fetcher, a program that fetches a page only when a person asks for it, for example with a question or a shared link. According to Anthropic, all three bots honor the rules in a robots.txt file, and site owners can allow or limit each one separately.
Claude-SearchBot in robots.txt
A dedicated group in robots.txt controls Claude-SearchBot specifically: a block of lines starting with “User-agent: Claude-SearchBot” and followed by its rules. A “Disallow: /” rule in that group bars it from every page of the site; with a path such as “Disallow: /customer-portal/”, only that area is off-limits to it. Anthropic’s documentation shows the full-site block (“Disallow: /”) using ClaudeBot as the example. If the file has no group for Claude-SearchBot, the rules under “User-agent: *”, which address every crawler without a group of its own, apply to it; a “Disallow: /” in that group then blocks Claude-SearchBot as well.
What blocking Claude-SearchBot means for visibility in Claude
If a site blocks Claude-SearchBot, Anthropic says this prevents its system from indexing the site’s content for web search. According to Anthropic, that may reduce the site’s visibility and accuracy in Claude’s web search. The answers affected are the ones in which Claude cites sources: according to Anthropic, every answer that uses web search includes citations, that is, AI citations. Conversely, allowing access does not guarantee a citation, because Anthropic does not disclose how Claude selects the sources for an answer.
Blocking Claude-SearchBot does not necessarily remove a site from Claude altogether. Claude-User can still fetch a page when someone asks Claude a question, as long as robots.txt does not block it too. According to Anthropic, Claude’s web search also draws on partners that index web pages with their own bots. For them, Anthropic names a separate control: the noindex directive in a page’s HTML tells these partners not to index the page and therefore not to supply it to Claude. And what Claude’s language models learned in training can flow into answers as parametric knowledge.
Deliberately keeping content out of AI answers is known as an AI answer opt-out. For businesses that want to appear as a source in Claude’s answers, allowing Claude-SearchBot is the natural choice.
Firewalls and bot protection
Whether Claude-SearchBot can reach a website depends on more than robots.txt. Firewalls, hosting, or a content delivery network (CDN)—a service that delivers websites through many servers—can also turn away automated requests; these settings are known as bot management. According to Anthropic, its bots respect technical protections against automated access and, for example, do not try to bypass CAPTCHAs, the challenges meant to tell people from programs. Such a challenge therefore keeps Claude-SearchBot out even if robots.txt allows it.