Trends · Sites with an AI crawler policy
Sites with an AI crawler policy across the web
The share of sites where a robots.txt names an AI crawler, either to allow it or to block it. It does NOT feed the index: deciding is not the same as being ready, and scoring it would make a value judgement out of a measurement.. It has gone up since May 2026, by more than the margin allows us to call a wobble — though last month on its own did not move.
Very poor9.6%
Sites with an AI crawler policy in August 2026
= −0.3 pp8.9%9.6%
What this measures
| Dimension | Weight in the index |
|---|---|
| 1 · Read | 53% of the score |
| 2 · Guide | 27% of the score |
| 3 · Act | 20% of the score |
| 4 · Reachable | multiplies the rest |
Who moved
| Segment | July 2026 | August 2026 | Movement |
|---|---|---|---|
| Act-On | 0.0% | 100.0% | ▲ +100.0 pp |
| Blue | 100.0% | 0.0% | ▼ −100.0 pp |
| Carrot quest | 0.0% | 100.0% | ▲ +100.0 pp |
| DonorPerfect | 0.0% | 100.0% | ▲ +100.0 pp |
| GemPages | 100.0% | 0.0% | ▼ −100.0 pp |
| Gomag | 100.0% | 0.0% | ▼ −100.0 pp |
| GTranslate app | 0.0% | 100.0% | ▲ +100.0 pp |
| Invoca | 0.0% | 100.0% | ▲ +100.0 pp |
| Kount | 100.0% | 0.0% | ▼ −100.0 pp |
| Mews | 100.0% | 0.0% | ▼ −100.0 pp |
| Nativo | 0.0% | 100.0% | ▲ +100.0 pp |
| Recharts | 100.0% | 0.0% | ▼ −100.0 pp |
| SeQura | 100.0% | 0.0% | ▼ −100.0 pp |
| Volusion | 0.0% | 100.0% | ▲ +100.0 pp |
| Wurfl | 100.0% | 0.0% | ▼ −100.0 pp |