ClaudeBot: what Anthropic's crawler does and how to control it
ClaudeBot is Anthropic's training crawler, not Clawdbot. How it differs from Claude-SearchBot and Claude-User, and how to allow or block each.
ClaudeBot is Anthropic’s web crawler. It collects public web content that could contribute to training Anthropic’s Claude models, and you control it with the ClaudeBot token in your robots.txt.
It is not Clawdbot. That name belonged to an unrelated open-source project, covered at the end of this post. This guide sticks to Anthropic’s own documentation: what ClaudeBot does, how it differs from Anthropic’s two other bots, how to allow or block each one, and how to see their visits in Measure.
What is ClaudeBot?
Anthropic describes ClaudeBot in its help article on how it crawls the web and how site owners can block the crawler. In Anthropic’s words, ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”
So ClaudeBot is the training crawler. It isn’t fetching your page because someone asked Claude about it right now. It’s gathering content that may end up in future training data.
If you restrict ClaudeBot, Anthropic says that “signals that the site’s future materials should be excluded from our AI model training datasets.” Note the word future. Blocking it is about what gets collected from now on.
What are Claude-SearchBot and Claude-User?
Anthropic runs three bots, and the same help article lists all of them. Each does a different job, so each can be allowed or blocked on its own.
Table: Anthropic’s three bots, per Anthropic’s help center
| robots.txt token | What it does | What blocking it does |
|---|---|---|
ClaudeBot | Collects web content that could contribute to model training | Signals your future content should be excluded from training datasets |
Claude-SearchBot | Crawls to improve the relevance and accuracy of Claude’s search results | Stops your content being indexed for search, which may reduce your visibility in Claude’s search results |
Claude-User | Fetches pages when a Claude user asks a question | Stops Claude retrieving your pages for a user’s query, which may reduce your visibility in user-directed web search |
The short version:
- ClaudeBot is about training.
- Claude-SearchBot is about search indexing.
- Claude-User is a person, right now, asking Claude something that needs your page.
That split matters. Plenty of sites are happy to be read when a user asks, and happy to be found in search, but would rather not be in training data. Anthropic’s setup lets you make that call per bot.
What do the user agents look like?
The names above are the tokens you put on a User-agent: line in robots.txt. Anthropic’s help article uses ClaudeBot this way in its own examples. Under the robots.txt standard, RFC 9309, crawlers must match that line case-insensitively, so claudebot works too.
User agent strings can be faked. Anthropic publishes a list of the IP addresses its crawlers use at claude.com/crawling/bots.json. If a request claims to be ClaudeBot and its source IP is on that list, Anthropic says it came from Anthropic.
How do I block ClaudeBot in robots.txt?
Add this to the robots.txt file in your site’s top-level directory. This is Anthropic’s own example:
User-agent: ClaudeBot
Disallow: /Anthropic asks you to do this for every subdomain you want to opt out. docs.example.com needs its own robots.txt.
Use robots.txt rather than blocking IP addresses. Anthropic says IP blocking “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” If ClaudeBot can’t read your robots.txt, it never learns that you said no.
How do I slow ClaudeBot down instead?
If the problem is load rather than training, Anthropic supports the non-standard Crawl-delay extension. Its example:
User-agent: ClaudeBot
Crawl-delay: 1Anthropic also says its bots honor “do not crawl” signals in robots.txt and won’t try to bypass CAPTCHAs.
How do I allow or block each Anthropic bot?
Give each bot its own group. Here’s one common setup: keep out of training, stay in Claude’s search and in answers to users.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /A few rules from RFC 9309 that help here:
- No rule means allowed. If no group matches a crawler and there’s no
*group, no rules apply. You don’t strictly need theAllow: /lines, but they make your intent obvious. - The most specific rule wins. You can block a folder and allow one page inside it. When an
Allowand aDisallowmatch equally,Allowis used. - Groups for the same bot combine. If two groups match one crawler, their rules are merged.
To block all three, give each one Disallow: /. Weigh what you lose first. Per Anthropic, blocking Claude-SearchBot and Claude-User may reduce how visible your site is when people search or ask Claude.
How do I see ClaudeBot and Claude-User visits in Measure?
Measure counts humans, AI agents and bots separately. AI agents are their own group: assistants fetching a page for someone, like Claude-User, and crawlers that train models, like ClaudeBot. Each is named with its operator and purpose, and traced to the pages it read.
The Measure team confirmed how Anthropic’s three bots are classified:
- ClaudeBot: AI agent, Anthropic, training
- Claude-SearchBot: AI agent, Anthropic, search
- Claude-User: AI agent, Anthropic, assistant
None of them count as human visitors, and none get lumped in with uptime checks or SEO tools.
If you use an AI assistant, you can also ask. Measure’s MCP server has a get_agents tool that returns every AI agent that fetched your site, with its operator, purpose, request count and top page. get_breakdown can split traffic by agent. So you can ask “how often did ClaudeBot visit this week, and what did it read?” and get the real numbers back.
For a wider look at what to check first, see how to get value from Measure in your first week and 5 things you can do with Measure and Claude.
Is ClaudeBot the same as Clawdbot or OpenClaw?
No. According to Wikipedia’s OpenClaw entry, OpenClaw (formerly Clawdbot, then Moltbot) is an open-source AI agent built by Peter Steinberger. It isn’t affiliated with Anthropic. If you’re searching for crawler settings, ClaudeBot is the one you want.
FAQ
Does blocking ClaudeBot remove my site from Claude?
No. Blocking ClaudeBot signals that your future content should be excluded from training. Claude-SearchBot and Claude-User are separate bots with their own tokens. Block those too if you want Claude to stop indexing or fetching your pages.
Does blocking ClaudeBot remove content it already collected?
Anthropic’s help article talks about future materials being excluded from training datasets. It doesn’t say blocking removes anything collected before.
Should I block ClaudeBot by IP address?
Anthropic recommends against it. IP blocking can stop ClaudeBot reading your robots.txt, so your opt-out may not register. Use robots.txt, and use Anthropic’s IP list to check whether a request really came from Anthropic.
Do I need a separate robots.txt for each subdomain?
Yes. Anthropic asks you to add the rules for every subdomain you want to opt out.
Will Measure count ClaudeBot as a visitor?
No. Measure counts ClaudeBot as an AI agent, apart from your human visitors, so a crawl of your docs doesn’t inflate your visitor numbers.
See who’s reading your site
Measure shows you the humans, the AI agents and the bots on your site, each counted on its own, with no cookies and no consent banner. It’s $29 a month for up to 40 sites, with a 14-day trial. Start your 14-day trial or see pricing.
Let your agent read the numbers.
$29 a month covers 40 sites, then $0.50 a site. 14-day trial.