AI bots: what each one does, when to block it and what you lose

Since September 15, 2026, the decision to let bots in or keep them out, which used to happen by omission, has a default set by third parties.

Updated on September 28, 2026. On the 15th itself, Cloudflare published a setting that separates blocking training from search visibility, and it changes the way out of the multipurpose crawler trap. What changed is in the section on the Accountable category, right after the Google case.

The four AI bot functions that get mixed up

The public discussion treats AI bots as a single category, and the effect of blocking each type is completely different. Four functions need to be separated before any decision.

Function	What the bot does	What you lose by blocking it
Training	Collects content to train models. The use is future and diffuse.	Nothing immediate in traffic. You lose a share of the corpus that shapes the model’s knowledge.
Search and indexing	Builds the index that feeds answers with citations, much like a search engine.	Citations and links in generated answers. This is the most expensive loss for anyone who wants visibility.
Agent on request	Fetches the page when a person asks the assistant for something.	Access by your own audience when they use an assistant to reach you.
Dual function	The same user agent feeds traditional search and AI features.	Presence in traditional search along with AI, because the block hits both.

Who is who, bot by bot and operator by operator

Deciding whether to block or allow only starts to make sense once you know which agent does what, and identification is usually the step people skip. The same operator runs different crawlers with different purposes, and the agent name does not always tell you which is which.

Operator	User agent	Category
OpenAI	GPTBot	AI Crawler, training
OpenAI	OAI-SearchBot	AI Search, indexing for citation
OpenAI	ChatGPT-User	AI Assistant, fetch on request
Anthropic	ClaudeBot	AI Crawler, training
Anthropic	Claude-SearchBot	AI Search
Anthropic	Claude-User	AI Assistant
Perplexity	PerplexityBot	AI Search
Perplexity	Perplexity-User	AI Assistant
Google	Googlebot	Search engine
Google	Google-CloudVertexBot	AI Crawler, at the site owner’s request
Microsoft	bingbot	Search engine
Apple	Applebot	AI Search
Meta	meta-externalagent	AI Crawler
Meta	meta-externalfetcher	AI Assistant
Meta	FacebookBot	AI Crawler
ByteDance	Bytespider	AI Crawler
Common Crawl	CCBot	AI Crawler
Amazon	Amazonbot	AI Crawler
DuckDuckGo	DuckAssistBot	AI Assistant
Mistral	MistralAI-User	AI Assistant

The table is a starting point, and it ages. Operators launch new agents, rename old ones and change the purpose of existing ones without public notice. Checking the Cloudflare reference before writing any rule is the step that keeps you from blocking the wrong thing.

The multipurpose crawler trap

An agent that declares a single function is easy to handle. The difficulty shows up with the one that bundles two functions under the same identity, because a rule written for one of them hits the other as well, with nothing on the dashboard to warn you.

The consequence is asymmetric, and that is what makes the mistake expensive. Blocking the agent that also does search removes the site from answers with links, which is exactly the surface worth occupying, and it does not stop collection by anyone who ignores the directive. The part you care about disappears and the part you wanted to avoid stays.

The Google case, which almost everyone gets wrong

The most common mistake on this list comes from a reasonable assumption: that Google has a specific crawler for artificial intelligence, and that blocking it settles the training question without touching search. The documentation describes a different arrangement.

The difference between the two controls is one of layer. One deals with how content is used after collection, the other with how it is displayed in results. Treating them as synonyms is what produces the block that removes the brand from the answer while protecting nothing. The practical rule is to write each directive knowing which of the two things it governs, and to check the effect in Search Console afterwards instead of guessing from the agent name.

What changed on September 15: the Accountable category

The new setting solves part of the problem described in the two previous sections, with one condition you need to understand before switching it on.

For anyone stuck in the multipurpose crawler trap, this creates a documented route: the no-training preference goes into robots.txt, and Googlebot, Bingbot and Applebot keep indexing. The condition lies in the entry criterion for the list, which is curated by a vendor. An operator that bundles search and training under the same identity and did not receive the designation loses both when the setting is switched on.

When to block each type of bot, by business

There is no single answer, because what is at stake changes with the revenue model. A site that lives on advertising loses money when the answer is given without a visit. A site that sells a complex service can gain by appearing in the answer even without a click, because the decision happens later and somewhere else.

Type of business	Training	Search and agent
Ad-supported media	Block. The content is the product and the return from training is zero.	Allow search. Evaluate agents according to your paywall policy.
E-commerce	Allow or block according to your catalog policy.	Allow both. Citations in answers and shopping agents are a sales channel.
SaaS and services	Blocking training makes little practical difference.	Allow both. Citation is the main discovery gain.
Consulting and personal brand	Allow. Being in the corpus helps the model remember you.	Allow both, no exceptions.
Paid or subscription content	Block.	Allow search only on open pages, with the rest protected by authentication.

The table organizes the decision by type of business and is meant to open the conversation. The final choice depends on two numbers only the client has: how much revenue depends on a visit, and how much depends on being considered before the visit.

A robots.txt that separates training from search

The example below applies the separation from the table to a site that wants out of model training while still appearing in search and in answers with links. The names are the ones operators document today and they change over time, so the list needs to be checked against the Cloudflare reference before going to production.

# Model training: blocked (decision recorded on YYYY-MM-DD)
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Search in AI systems: allowed
User-agent: OAI-SearchBot
Allow: /

Two notes on the example. Google-Extended appears as a control token, with no crawler of its own, as shown in the section on Google. And access that a person triggers from inside the assistant follows a different rule, described by the operator itself:

That is why it stays out of the file. Writing a rule for it would suggest a block that the documentation does not guarantee.

The blocking mistake I find most often in audits

I do not trust a robots.txt with forty lines of exceptions. Whoever writes forty lines of polite requests usually has not opened the logs to see who walked in ignoring all forty.

The mistake is the blanket block of everything with an AI-sounding name in robots.txt, written in one afternoon, without separating training from search. The result is a brand that drops out of generated answers and stays in the corpus of the models that ignore robots.txt, which combines the worst of both scenarios.

The second most common mistake is the opposite and comes from omission: leaving everything open without knowing who comes in, and finding out only when the bandwidth cost shows up on the invoice.

Content Use Signals in robots.txt

The same announcement brought a second mechanism, less discussed and more interesting in the medium term. Instead of allowing or blocking access, it declares what can be done with the content after access has already happened.

It is worth implementing even with partial adoption, for two reasons. The cost is one line in robots.txt, and a recorded statement has potential legal value that omission lacks. What does not hold up is presenting it to the client as an effective block.

llms.txt usually enters the same conversation, as if it were another kind of signal for AI systems. Its actual status is more modest than the way it circulates.

Publishing it costs little and works as a map for whoever decides to read it. It offers no access control, and no effect on citation had been publicly measured as of this update.

What to check after September 15

• Find out which bots access your site today, using access logs or your CDN’s AI crawler control panel.
• Split the decision in two: training is an intellectual property choice, search and agents are a distribution choice.
• Check whether your domain falls under the new default, which depends on being a new domain on Cloudflare and on displaying ads.
• Before switching on any training block, check whether it reaches a dual-function crawler, and treat Googlebot, Applebot and Bingbot as a separate case.
• Evaluate the Disallow AI Training setting and check whether any crawler that matters for your distribution was left off the Accountable list.
• Document the decision in robots.txt with a dated comment, so the next person understands the intent without having to guess.

---

References

Each source was checked on the access date shown.

1. Article

   Your site, your rules: new AI traffic options for all customers

   Cloudflare Blog · Jul 1, 2026

   Copy citation

2. Official documentation

   Bot reference

   Cloudflare AI Crawl Control docs

   Copy citation

3. Official documentation

   List of Google's common crawlers

   Google for Developers

   Copy citation

4. Official documentation

   AI features and your website

   Google Search Central

   Copy citation

5. Article

   Have it both ways: stay discoverable in search while disallowing AI training

   Cloudflare Blog · Sep 15, 2026

   Copy citation

6. Official documentation

   Overview of OpenAI Crawlers

   OpenAI

   Copy citation

7. Official documentation

   The /llms.txt file

   HOWARD, Jeremy · llmstxt.org · Sep 3, 2024

   Copy citation