Anthropic introduced Claude in March 2023. The AI assistant runs in the browser and as a desktop and mobile app, and it can be built into other applications through an application programming interface (API). When people ask Claude about providers, products, or services, it writes its own answer. Your business can appear in that answer as a mention in the text or as a linked source. Both shape your AI visibility.
When Claude retrieves content from the web
Without web access, Claude answers from what its language models learned in training. That knowledge only extends to a certain date, its knowledge cutoff, which Anthropic lists for each model. With its web search feature, Claude can also retrieve current content from the web. According to Anthropic’s API documentation, Claude does so when a request depends on information that is current, changing, or outside its training data, for example:
- recent events, news, or announcements
- current prices, rates, or statistics
- information about specific organizations, people, or products that might have changed
- explicit requests to look something up
When a request concerns established facts, is a creative task, or refers to text already in the conversation, Claude answers directly instead. For businesses, the third item matters most: questions about organizations and products whose details might have changed are explicitly among the reasons Claude retrieves web content. What your website says about your offering, prices, or locations can thus flow into the answer, which is why it pays to keep those details current there (content freshness). For pages that web search returns, the API can also report when each was last updated. Anthropic does not disclose how Claude weighs that date.
With web search turned on, Claude can also read a specific page directly when someone enters its address in the chat. Anthropic calls this web fetch. When Claude is built into other applications through the API, their providers decide whether Claude may use web search there.
How Claude cites sources
When Claude uses web content, its answer includes direct citations, links for further reading, and relevant quotes where appropriate, according to Anthropic. These citations are AI citations: they lead from the answer to the page in question. In the API, citations are always enabled for web search, and each one includes a short excerpt of the cited passage along with the page’s title and URL. How well such a passage works on its own is a matter of its citability. Anthropic requires applications that display Claude’s answers directly to show the citations as well. The link to a cited page can therefore also appear in other providers’ applications.
Anthropic’s crawlers
Anthropic runs three crawlers (programs that fetch web pages automatically) and says all of them honor the rules in a robots.txt file: ClaudeBot collects content for training Anthropic’s models, Claude-SearchBot gathers content to improve Claude’s web search, and Claude-User fetches pages when someone asks Claude a question. As long as Claude-SearchBot and Claude-User are allowed, Claude can use your pages for its answers, regardless of what you decide about ClaudeBot and thus about training.