GPTBot vs OAI-SearchBot vs ChatGPT-User: which AI agents to allow

GPTBot collects content to train OpenAI’s models, OAI-SearchBot indexes pages for ChatGPT’s search features, and ChatGPT-User fetches a page when a person asks ChatGPT something. You can block one and allow the others, because OpenAI treats each robots.txt setting separately.

The same split, training versus search versus assistant, now shows up at Anthropic, Perplexity and Meta. Here is what each operator says about its own agents, a robots.txt you can copy, and how to see which ones actually visit you.

What do OpenAI’s three agents do?

OpenAI documents all three on its crawlers page:

• GPTBot is “used to make our generative AI foundation models more useful and safe”. This is the training crawler.
• OAI-SearchBot is used “to surface websites in search results in ChatGPT’s search features”.
• ChatGPT-User is used “for certain user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent.”

The key line for site owners: “Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.”

Table: OpenAI’s agents, from OpenAI’s crawlers page (checked 2026-10-28)

On ChatGPT-User, OpenAI says it “is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI also notes that for search results “it can take ~24 hours from a site’s robots.txt update for our systems to adjust.”

OpenAI publishes IP ranges for each: gptbot.json, searchbot.json and chatgpt-user.json.

How do Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User differ?

Anthropic uses the same three-way split. From its help center article on crawling:

• ClaudeBot collects “web content that could potentially contribute to their training.” Restricting it “signals that the site’s future materials should be excluded from our AI model training datasets.”
• Claude-SearchBot “navigates the web to improve search result quality for users.” Disabling it “prevents our system from indexing your content for search optimization.”
• Claude-User supports people using Claude: “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Disabling it “prevents our system from retrieving your content in response to a user query.”

Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”, and that rules apply per subdomain. It also supports Crawl-delay. It advises against IP blocking, since that can stop the bots reading your robots.txt, and lists its IPs at claude.com/crawling/bots.json. The article names each agent but doesn’t give full user agent strings, so we don’t either.

What about Perplexity?

Perplexity runs two agents, documented on its bots page:

• PerplexityBot (Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)) is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity tells you to allow it in robots.txt if you want to appear in its results.
• Perplexity-User (Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)) visits a page when someone asks Perplexity a question. Perplexity says it “generally ignores robots.txt rules” because a user asked for the page.

There is no Perplexity training crawler on that page. IP ranges are at perplexitybot.json and perplexity-user.json.

What are Meta-ExternalAgent and Bytespider?

Meta documents its agents on its web crawlers page. Meta-ExternalAgent “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Meta lists its user agent as meta-externalagent/1.1 and shows robots.txt rules written for meta-externalagent. Meta’s user-initiated fetcher, Meta-ExternalFetcher, is a separate agent that Meta says may disregard robots.txt.

Bytespider is ByteDance’s crawler. We couldn’t open a ByteDance page documenting it, so we won’t quote a user agent string, a purpose or its robots.txt behaviour here. What we can tell you is how Measure files it: as a ByteDance agent with a training purpose. If you add a robots.txt rule for it, check your agent traffic afterwards to see whether the visits stop.

How do they all compare?

Table: AI agents by purpose (operators’ own pages, checked 2026-10-28)

The pattern: training crawlers are the ones you can reliably opt out of with robots.txt. Assistant agents act for a person, and some operators say robots.txt may not stop them.

What robots.txt allows search and assistants but blocks training?

This keeps you visible in ChatGPT, Claude and Perplexity answers and search, and opts out of training for the three operators that document a training crawler:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Allow search and assistant agents
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

To block everything from one operator instead, list all of its tokens under Disallow: /. Remember Anthropic’s note that each subdomain needs its own file, and OpenAI’s note that search changes take about a day.

How do you see each agent in Measure?

Measure counts humans, AI agents and bots separately, and names each agent with its operator and purpose (assistant, training or search). GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bytespider and Meta-ExternalAgent are all on its agent list. Agent traffic is always broken out from human traffic, so a crawler reading your docs doesn’t inflate your visitor count.

There is one catch. Most AI crawlers don’t run JavaScript, so a JS snippet misses them. The snippet still catches the agents that do render pages, but for the full picture, capture visits on your server too. Your server sends POST https://lets.measure.events/api/collect with your site_id and the path, plus the crawler’s user agent or an explicit agent name. Measure classifies it the same way it classifies a snippet visit. The For agents page documents the endpoint.

Then ask. Measure’s MCP server has a get_agents tool that lists every AI agent that fetched your site, with its operator, purpose, request count and top page. Ask your assistant “Did GPTBot stop visiting after I changed robots.txt?” and you get an answer from your own data. More on that in Ask your AI agent about your traffic.

FAQ

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says each setting is independent. Search uses OAI-SearchBot, so you can disallow GPTBot and still allow OAI-SearchBot.

Can I block ChatGPT-User with robots.txt?

You can add the rule, but OpenAI says that because these visits are user-initiated, “robots.txt rules may not apply.” Perplexity says Perplexity-User generally ignores robots.txt.

Should I block ClaudeBot?

That’s your call. Anthropic says blocking ClaudeBot excludes your future content from training. Claude-SearchBot and Claude-User are controlled separately, so blocking ClaudeBot alone doesn’t stop Claude finding or reading your pages for users.

Will Measure count these visits as visitors?

No. AI agents and bots are counted separately from humans.

See which agents read your site

Robots.txt says what you want. Your traffic shows what happened. Measure is $29 a month for up to 40 sites, with a 14-day trial. Start your 14-day trial or see pricing.