AI agent · August 2026
Claude-Web in robots.txt: who blocks it
1.0% of the 15,009 sites in the census name Claude-Web in their robots.txt, and 71.9% of those block it.
Sites naming it
- Sample
- n=15,009
- Coverage
- 100.0%
The August 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 0.1% sample of the corpus as every other figure on this site — 15,009 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.
Sites blocking it
- Sample
- n=15,009
- Coverage
- 100.0%
What Claude-Web actually does
Run by Anthropic. An older token for fetching a page on a user’s behalf. Blocking it means that when someone asks about your page, the assistant cannot open it.
What to write in your robots.txt
Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.
User-agent: Claude-Web
Disallow: /User-agent: Claude-Web
Allow: /