Agentic Web Census

August 2026by Omitsis

English

AI agent · August 2026

anthropic-ai in robots.txt: who blocks it

3.2% of the 15,009 sites in the census name anthropic-ai in their robots.txt, and 89.7% of those block it.

3.2%
Sites naming it

Sites naming it

Sample
n=15,009
Coverage
100.0%

The August 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 0.1% sample of the corpus as every other figure on this site — 15,009 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.

2.9%
Sites blocking it

Sites blocking it

Sample
n=15,009
Coverage
100.0%
Name it
Name it: 3.2%
3.2%
Block it
Block it: 2.9%
2.9%
00.250.500.751.0
Of the 15,009 sites in the census, 3.2% name anthropic-ai and 2.9% forbid it. Both bars are the same base, so the gap between them is the sites that named it to let it in.

What anthropic-ai actually does

Run by Anthropic. An older token for the same crawling. Still widely written in robots.txt files, so it is still worth counting. Blocking it costs you no traffic: this crawler does not send anyone your way.

What to write in your robots.txt

Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.

User-agent: anthropic-ai
Disallow: /
robots.txtTurns anthropic-ai away. Blocking it costs you no traffic: this crawler does not send anyone your way.
User-agent: anthropic-ai
Allow: /
robots.txtLets it through, said out loud rather than by saying nothing. Either way it is a published preference and not a wall: robots.txt enforces nothing.