Who blocks AI crawlers? robots.txt rules of the Cloud 100 (2026 study)

Should you block GPTBot? The companies that build and sell AI software have mostly answered that already, and the answer is no. We fetched the robots.txt of every company on the 2025 Forbes Cloud 100 on October 28, 2026 and parsed the rules for fifteen AI crawlers. Of the 96 files we could read, 89 block no AI crawler at all. Only 2 block OpenAI’s training crawler, GPTBot, and not one blocks a search or answer bot such as OAI-SearchBot, PerplexityBot, Claude-SearchBot or ChatGPT-User.

The handful who do block are aiming at training crawlers, and some of them carve out exceptions for the pages they want quoted. Everything below comes from that crawl. The full dataset is downloadable as a CSV.

Key findings

• 89 of 96 (93%) Cloud 100 sites with a readable robots.txt block no AI crawler at the site root.
• 0 of 96 block a search or user-fetch bot: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot or Claude-User.
• 2 of 96 block GPTBot (Canva and Midjourney). 1 blocks ClaudeBot (Canva). 2 block Google-Extended (Midjourney and Together AI).
• The most-blocked crawlers are the ones with the weakest “send traffic back” story: CCBot, Bytespider and meta-externalagent, 4 sites each.
• 21 of 96 (22%) mention at least one AI crawler by name. 14 of those 21 name them only to let them in.
• 94 of 96 list a sitemap in robots.txt, the most basic crawl hint there is.

Why “search bot” and “training bot” are different decisions

The single most useful thing to know before editing robots.txt is that each AI company now runs several crawlers, and they do different jobs. Blocking one does not block the others.

Sources for the descriptions: OpenAI’s crawler documentation, Anthropic’s crawler help article, Perplexity’s bot guide and Google’s common-crawlers reference and Common Crawl’s CCBot page.

Read the table as two questions. Do you want to appear when someone asks an AI assistant about your category? Then the search and user-fetch bots (the zeros in the right-hand column) have to be able to reach you. Do you want your content used to train the next model? That is GPTBot, ClaudeBot, Google-Extended, CCBot and friends, and it is a separate choice with separate trade-offs. The Cloud 100 have overwhelmingly said yes to the first. Most have not bothered to say anything about the second.

The seven sites that do block something

The exceptions are more interesting than the rule, because each one shows a deliberate policy rather than a default:

1. Canva blocks GPTBot, ClaudeBot, anthropic-ai, Applebot-Extended, CCBot, Bytespider and meta-externalagent at /, then, in the same group, re-opens /create/ and its localized equivalents (/de_de/erstellen/, /fr_fr/creer/ and so on). It is the clearest example of a “train on our marketing pages, not our templates” policy we found.
2. Midjourney blocks the widest set: GPTBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot.
3. Calendly blocks CCBot at the root but allows two paths: /llms.txt and /pricing. If a model is going to learn anything about Calendly from Common Crawl, it will be the price list.
4. Forter blocks CCBot, Bytespider and meta-externalagent under a comment that reads “Scrapers and content harvesters are disallowed.”
5. Notion blocks Amazonbot and meta-externalagent (along with SEO tools such as SemrushBot).
6. Postman blocks Bytespider only.
7. Together AI blocks Google-Extended only, with a comment saying the intent is to “block AI training only”.

Notice what none of them do: block the bots that put a link to them in front of a person asking a question.

What this means for your robots.txt

If you sell to businesses, the companies on this list are a reasonable benchmark: they are large, technical and have legal teams that think about data use. Their revealed preference suggests a default that we also recommend to clients:

• Allow every search and user-fetch bot. Blocking OAI-SearchBot removes you from ChatGPT search answers, by OpenAI’s own description. There is no upside for a company that wants to be found.
• Decide on training crawlers deliberately, and write the decision down. If your content is your product (a publisher, a template library, a dataset), blocking GPTBot, ClaudeBot, Google-Extended and CCBot is defensible. If your content exists to market a product, being in the training data is arguably free brand exposure. Either way, the choice belongs to someone senior, not to whoever last edited the file.
• Use path rules, not all-or-nothing. Canva and Calendly show the pattern: block at /, then Allow: the pages you want models to know about.
• Do not assume robots.txt is the whole picture. A CDN-level bot rule can block a crawler before it ever reads robots.txt. Four Cloud 100 sites (Gusto, Cohesity, Personio and Axonius) returned 403 or 429 to our ordinary fetch of the file itself, which is exactly the kind of block a robots.txt audit cannot see.

A minimal robots.txt that welcomes AI search while opting out of training looks like this:

# AI search and answer engines: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If you want to know whether AI engines are actually citing you after you open the door, the measurement side is covered in our guide to tracking AI visibility and measuring AI search traffic in GA4. Getting crawled is step zero of generative engine optimization; our GEO and AI search service starts with exactly this audit.

Methodology

1. Sample. All 100 companies on the 2025 Forbes Cloud 100, taken from Forbes’ published list. Where Forbes listed no website or an outdated one, we used the company’s current primary domain (for example cursor.com for Anysphere and gomotive.com for Motive).
2. Fetch. On 2026-10-28 we requested each homepage, followed redirects to the final origin, and fetched /robots.txt from that origin with a standard desktop Chrome user-agent string.
3. Parse. We grouped rules by User-agent per RFC 9309: a crawler uses the most specific group that names it, and falls back to * only if none does. A crawler counts as blocked if the longest matching rule for the path / is a Disallow.
4. Exclusions. Four sites returned 403 or 429 for robots.txt, leaving 96 readable files. Path-level blocks that leave / open (for example blocking only /app/) are not counted as blocks.
5. Limits. robots.txt states intent; it does not enforce anything, and some user-initiated agents say they may not follow it. Network-level blocking is invisible to this method. One crawl is one snapshot, so rules may have changed since.

FAQ

Should I block GPTBot in 2026? Only if you do not want your content used to train OpenAI’s models. GPTBot is OpenAI’s training crawler; blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot. In our study of the Forbes Cloud 100, only 2 of 96 readable robots.txt files block GPTBot.

Does blocking GPTBot remove my site from ChatGPT? No. OpenAI documents separate crawlers: GPTBot for training, OAI-SearchBot for ChatGPT search results, and ChatGPT-User for user-initiated visits. To leave ChatGPT search you would have to block OAI-SearchBot, which none of the Cloud 100 do.

Does Google-Extended affect my Google rankings? No. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” It controls whether your content is used for Gemini training and grounding.

Is robots.txt enough to stop AI crawlers? It is a request, not a lock. Well-behaved crawlers follow it, user-initiated fetchers may not, and the only hard block is at the server or CDN. That is also why a robots.txt audit should be paired with a check of your firewall’s bot rules.

Related reading:

• Cloud 100 llms.txt study: 62% of the same companies publish an llms.txt file
• llms.txt explained: how to write one in an hour
• GEO vs SEO: what changes when the reader is a model

Sources: Forbes Cloud 100 (2025); OpenAI crawler docs; Anthropic crawler docs; Perplexity bot guide; Google common crawlers; RFC 9309. Dataset: cloud100-website-study-2026.csv.