Good bots vs bad bots on your website: how to tell them apart

A good bot is automated traffic that says who it is, follows your robots.txt, can be verified against its operator’s published DNS names or IP ranges, and gives you something back, like search listings or citations. A bad bot hides or fakes its identity, ignores your rules, and takes without returning anything: scrapers, credential stuffers and anything wearing someone else’s user agent.

Most sites get both, every day. Here’s how to tell them apart, and what each does to your analytics.

What makes a bot good?

Four things, roughly in order of importance.

• It identifies itself. It sends a user agent that names the operator and links to a page explaining the bot. OpenAI’s GPTBot, for example, sends Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot (OpenAI).
• It follows robots.txt. Google says its common crawlers “always respect robots.txt rules for automatic crawls” (Google).
• You can verify it. The operator publishes a way to check that a request really came from them: reverse DNS, a list of IP ranges, or both.
• It gives something back. A search crawler sends you visitors. An assistant fetch answers a real person’s question about your page.

A bot that does all four is one you usually want. Whether you want every good bot is a separate choice, and robots.txt is where you make it.

What makes a bot bad?

The opposite of each point above.

• Scrapers copy your content or prices in bulk, often with a generic or fake user agent, and send nothing back.
• Credential stuffing is “the automated injection of stolen username and password pairs (“credentials”) in to website login forms, in order to fraudulently gain access to user accounts“ (OWASP).
• Spoofed user agents claim to be someone they’re not. Google is blunt about it: “The HTTP user agent string can be spoofed” (Google).

That last one matters most. A user agent is a claim, not proof. Anyone can send a request that says Googlebot.

What kinds of bots visit a website?

Six groups cover most of it.

Search crawlers

Googlebot is the one most sites care about. Its desktop user agent ends in (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36, and its robots.txt token is Googlebot (Google).

AI training crawlers

These collect content for model training. OpenAI says GPTBot is used “to crawl content that may be used in training our generative AI foundation models” (OpenAI). Anthropic says ClaudeBot collects “web content that could potentially contribute to their training” (Anthropic).

Google handles this differently. Google-Extended “doesn’t have a separate HTTP request user agent string”; it’s a robots.txt token that controls whether content is used for Gemini training and grounding, while crawling happens under Google’s existing user agents (Google). You won’t see it in a log.

AI search crawlers

These index pages so they can show up in AI answers. OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features” (OpenAI). PerplexityBot is “designed to surface and link websites in search results on Perplexity” and, per Perplexity, “is not used to crawl content for AI foundation models” (Perplexity). Anthropic’s Claude-SearchBot works to “improve search result quality for users” (Anthropic).

Block these and you drop out of those answers. Sites opted out of OAI-SearchBot won’t appear in ChatGPT search answers (OpenAI).

Assistant fetches on a user’s behalf

When someone asks ChatGPT, Claude or Perplexity about a page, the assistant may fetch it right then. That’s ChatGPT-User, Claude-User and Perplexity-User. Because a person triggered the fetch, robots.txt is treated differently: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores robots.txt rules” (OpenAI, Perplexity).

These are closer to a visit than a crawl. Someone wanted your page.

Monitoring and uptime bots

You probably set these up yourself: uptime checks, SEO audit tools, link checkers. They’re harmless, but they hit the same pages on a schedule and add up fast in raw request counts.

Scrapers

No name, no docs, no IP list. Or worse, a borrowed name. You find them by their behaviour, not their label.

How do you verify a bot is real?

Check the IP, not the user agent.

Google documents two ways (Google):

1. Reverse DNS. Look up the IP from your logs with host, check the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP.
2. IP ranges. Match the IP against Google’s published JSON files, such as common-crawlers.json and user-triggered-fetchers.json.

host 66.249.66.1
# then run host on the name it returns, and confirm you get 66.249.66.1 back

AI operators publish IP lists you can match against:

• OpenAI: https://openai.com/gptbot.json, https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json (OpenAI)
• Anthropic: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic” (Anthropic)
• Perplexity: https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json (Perplexity)

A request that claims GPTBot from an IP outside OpenAI’s list isn’t GPTBot.

Does robots.txt stop bad bots?

No. The standard itself says “These rules are not a form of access authorization” (RFC 9309). Good bots read it and comply. Bad bots read it and ignore it, or never read it at all.

Use robots.txt to tell well-behaved bots what you want:

User-agent: ClaudeBot
Disallow: /

That example is straight from Anthropic’s docs, which also note that blocking their IPs instead “may not work correctly or persistently guarantee an opt-out”. For bad bots, you need rate limits, login protection and IP checks at your server or edge.

How do bots show up in your analytics?

Two ways, and both are a problem.

They inflate your human numbers. If bots are counted as visitors, your traffic looks bigger and your conversion rate looks worse than it is. We covered how bad data skews decisions in why your analytics are lying to you.

Or they’re missing entirely. Most AI crawlers don’t run JavaScript, so a JS snippet misses them. Googlebot and some assistants do render JavaScript, so a snippet catches some automated traffic and not the rest. You end up with neither a clean human count nor a full picture of who’s reading your site.

How does Measure split humans, AI agents and bots?

Measure counts three groups separately:

• Humans: “People with a browser. Pages, referrers, countries and devices, without cookies or fingerprinting.”
• AI agents: “Assistants fetching a page for someone, like ChatGPT-User and Claude-User, and the crawlers that train models, like GPTBot and ClaudeBot.”
• Bots: “Uptime checks, SEO tools and scrapers. Kept out of your human numbers, still there if you want them.”

For each AI agent, Measure shows the operator and purpose (assistant, training or search), its requests and its top page (for agents). Measure sorts traffic by the user agent a request presents. Classic search crawlers like Googlebot and Bingbot land in bots, not AI agents.

To catch crawlers that never run your script, send requests from your server. The for-agents page documents POST https://lets.measure.events/api/collect with site_id, path and agent, and Measure classifies those the same way.

You can also ask about it in plain language. With the MCP server connected, get_agents lists the AI systems that visited and why. See ask your AI agent about your traffic.

FAQ

Is Googlebot a good bot?

Yes. It identifies itself, follows robots.txt for automatic crawls, and Google publishes how to verify it. Just verify before you trust the name, because the user agent can be spoofed.

Should I block AI crawlers?

It depends on the crawler. Training crawlers like GPTBot and ClaudeBot can be blocked in robots.txt. Blocking AI search crawlers like OAI-SearchBot or PerplexityBot can keep you out of those answers. Decide per bot, not all at once.

Are assistant fetches like ChatGPT-User bots or visitors?

Automated, but on a person’s behalf. Measure counts them as AI agents, separate from both humans and bots, so you can see them without them inflating your visitor numbers.

Can I see Google-Extended in my logs?

No. Google-Extended is a robots.txt token with no user agent string of its own.

See your humans, agents and bots separately

Measure is $29 a month for up to 40 sites, then $0.50 per site. Start your 14-day trial or see pricing.