AI agent · August 2026
ChatGPT-User in robots.txt: who blocks it
2.0% of the 15,009 sites in the census name ChatGPT-User in their robots.txt, and 71.0% of those block it.
Sites naming it
- Sample
- n=15,009
- Coverage
- 100.0%
The August 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 0.1% sample of the corpus as every other figure on this site — 15,009 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.
Sites blocking it
- Sample
- n=15,009
- Coverage
- 100.0%
What ChatGPT-User actually does
Run by OpenAI. Fetches a page because a person asked ChatGPT about it. Blocking it breaks that link for a reader who was already interested. Blocking it means that when someone asks about your page, the assistant cannot open it.
What to write in your robots.txt
Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.
User-agent: ChatGPT-User
Disallow: /User-agent: ChatGPT-User
Allow: /