AI agent · August 2026
Google-Extended in robots.txt: who blocks it
7.0% of the 15,009 sites in the census name Google-Extended in their robots.txt, and 92.6% of those block it.
Sites naming it
- Sample
- n=15,009
- Coverage
- 100.0%
The August 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 0.1% sample of the corpus as every other figure on this site — 15,009 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.
Sites blocking it
- Sample
- n=15,009
- Coverage
- 100.0%
What Google-Extended actually does
Run by Google. Controls whether your content is used to train Google’s generative models. It does NOT affect Google Search: blocking it leaves your search ranking alone. Blocking it costs you no traffic: this crawler does not send anyone your way.
What to write in your robots.txt
Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.
User-agent: Google-Extended
Disallow: /User-agent: Google-Extended
Allow: /