Which AI agents the web blocks

Agent	What it does	Sites naming it	Sites blocking it	Blocking, month by month
GPTBot GPTBot Operator OpenAI Of those naming it, blocked 88.7% Sample n=15,009 Crawls to train OpenAI models. Blocking it does not remove you from ChatGPT answers — that is OAI-SearchBot and ChatGPT-User, which are separate tokens.	Trains models	8.6%	7.6%	= +0.1 pp
ClaudeBot ClaudeBot Operator Anthropic Of those naming it, blocked 89.5% Sample n=15,009 Crawls to train Anthropic models.	Trains models	7.6%	6.8%	= +0.1 pp
CCBot CCBot Operator Common Crawl Of those naming it, blocked 95.5% Sample n=15,009 Builds the public Common Crawl corpus, which many model trainers use. Blocking it is an indirect way of opting out of several models at once.	Trains models	7.4%	7.1%	= +0.2 pp
Bytespider Bytespider Operator ByteDance Of those naming it, blocked 97.3% Sample n=15,009 Crawls to train ByteDance models.	Trains models	7.0%	6.8%	= +0.1 pp
Google-Extended Google-Extended Operator Google Of those naming it, blocked 92.6% Sample n=15,009 Controls whether your content is used to train Google’s generative models. It does NOT affect Google Search: blocking it leaves your search ranking alone.	Trains models	7.0%	6.5%	= +0.1 pp
Amazonbot Amazonbot Operator Amazon Of those naming it, blocked 97.3% Sample n=15,009 Crawls for Amazon’s assistant answers.	Indexes for answers	6.9%	6.7%	= −0.2 pp
meta-externalagent meta-externalagent Operator Meta Of those naming it, blocked 94.9% Sample n=15,009 Crawls to train Meta models.	Trains models	6.7%	6.4%	= +0.4 pp
Applebot-Extended Applebot-Extended Operator Apple Of those naming it, blocked 95.4% Sample n=15,009 Controls whether your content trains Apple’s generative models. Separate from Applebot, which indexes for Siri and Spotlight and is not affected.	Trains models	6.4%	6.1%	= +0.1 pp
anthropic-ai anthropic-ai Operator Anthropic Of those naming it, blocked 89.7% Sample n=15,009 An older token for the same crawling. Still widely written in robots.txt files, so it is still worth counting.	Trains models	3.2%	2.9%	▲ +0.5 pp
cohere-ai cohere-ai Operator Cohere Of those naming it, blocked 92.3% Sample n=15,009 Crawls to train Cohere models.	Trains models	2.7%	2.5%	▲ +0.6 pp
YouBot YouBot Operator You.com Of those naming it, blocked 95.5% Sample n=15,009 Indexes pages so You.com can cite them.	Indexes for answers	2.5%	2.4%	= +0.4 pp
PerplexityBot PerplexityBot Operator Perplexity Of those naming it, blocked 59.4% Sample n=15,009 Indexes pages so Perplexity can cite them with a link back.	Indexes for answers	2.3%	1.4%	= +0.1 pp
ChatGPT-User ChatGPT-User Operator OpenAI Of those naming it, blocked 71.0% Sample n=15,009 Fetches a page because a person asked ChatGPT about it. Blocking it breaks that link for a reader who was already interested.	Fetches on request	2.0%	1.4%	= +0.2 pp
OAI-SearchBot OAI-SearchBot Operator OpenAI Of those naming it, blocked 58.1% Sample n=15,009 Indexes pages so ChatGPT can cite them. This is the one to leave open if you want to appear in its answers.	Indexes for answers	1.8%	1.0%	= +0.3 pp
Claude-Web Claude-Web Operator Anthropic Of those naming it, blocked 71.9% Sample n=15,009 An older token for fetching a page on a user’s behalf.	Fetches on request	1.0%	0.7%	= −0.3 pp
Claude-User Claude-User Operator Anthropic Of those naming it, blocked 78.0% Sample n=15,009 Fetches a page because a person asked Claude about it.	Fetches on request	0.7%	0.5%	= +0.2 pp
Perplexity-User Perplexity-User Operator Perplexity Of those naming it, blocked 70.5% Sample n=15,009 Fetches a page because a person asked about it.	Fetches on request	0.6%	0.4%	= +0.2 pp