Agentic Web Census

August 2026by Omitsis

English

AI agent · August 2026

OAI-SearchBot in robots.txt: who blocks it

1.8% of the 15,009 sites in the census name OAI-SearchBot in their robots.txt, and 58.1% of those block it.

1.8%
Sites naming it

Sites naming it

Sample
n=15,009
Coverage
100.0%

The August 2026 crawl, parsed for a robots.txt rule naming this user-agent, on the same 0.1% sample of the corpus as every other figure on this site — 15,009 sites. It used to be measured on the whole corpus and this note used to say so; it moved to the monthly sample when the axis gained a series, so it now carries the same margin as everything else.

1.0%
Sites blocking it

Sites blocking it

Sample
n=15,009
Coverage
100.0%
Name it
Name it: 1.8%
1.8%
Block it
Block it: 1.0%
1.0%
00.250.500.751.0
Of the 15,009 sites in the census, 1.8% name OAI-SearchBot and 1.0% forbid it. Both bars are the same base, so the gap between them is the sites that named it to let it in.

What OAI-SearchBot actually does

Run by OpenAI. Indexes pages so ChatGPT can cite them. This is the one to leave open if you want to appear in its answers. Blocking it takes you out of that product’s answers, along with the link back to you.

What to write in your robots.txt

Put the rule in its own group. A Disallow under User-agent: * predates these agents and several of them do not treat it as applying to them — which is also why this census does not count it.

User-agent: OAI-SearchBot
Disallow: /
robots.txtTurns OAI-SearchBot away. Blocking it takes you out of that product’s answers, along with the link back to you.
User-agent: OAI-SearchBot
Allow: /
robots.txtLets it through, said out loud rather than by saying nothing. Either way it is a published preference and not a wall: robots.txt enforces nothing.