Training Bots vs Search Crawlers: The robots.txt Split That Decides Your AI Visibility

Quick answer: AI vendors now run separate crawlers for two different jobs. One set harvests pages to train models (GPTBot, ClaudeBot, Google-Extended). A second set indexes pages so AI search can cite them (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot). These are controlled by different user agents in your robots.txt, which means you can opt out of training while staying fully eligible for AI-search citations. The old advice to "block all AI bots" now backfires: it quietly deletes your brand from the fastest-growing referral channel of 2026. This article explains the split, lists the bots that matter, and gives you a copy-ready robots.txt policy.

Here is why this is urgent. A large share of site owners blocked AI crawlers in a single sweep, often through a hosting default or a well-meaning plugin, without realizing they were cutting off search indexers along with training scrapers. The result is a silent visibility loss: the pages still rank in Google, but ChatGPT, Perplexity, and Claude can no longer retrieve or cite them.

---

Why AI Vendors Split Their Crawlers in Two

Until recently, "an AI bot" was a single idea. You either let it in or you didn't. That mental model is now wrong, because the major labs have deliberately separated their crawlers by purpose.

The distinction that changes everything is purpose: a crawler that collects your pages for model training is a fundamentally different actor from one that indexes you for AI search answers. OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT search. Anthropic runs three separate bots. Google separates traditional search crawling from its Gemini training access. Because each purpose has its own user agent, each is a separate decision in your robots.txt.

This matters because the two activities have very different value to a publisher.

• Training crawls feed your content into the corpus a model learns from. Blocking this opts you out of training-corpus inclusion. For most publishers that benefit is aspirational anyway, since the labs disclose little about corpus selection and the direct visibility payoff is unclear.
• Search crawls decide whether your pages can be retrieved and cited when a user asks a question today. Blocking this removes you from that engine's answers entirely, right now, with a measurable traffic cost.

Collapse both into one "block all AI bots" rule and you trade a speculative training concern for a concrete loss of live citations. That is a bad trade for almost every commercial site.

---

The Economics: Why Search Citations Are Worth Protecting

The reason this decision has teeth is that AI search referrals are growing fast and converting well, while training crawls give back little in return.

Consider the imbalance in how often these bots take versus how often they send a visitor back:

• Anthropic's crawl-to-referral ratio peaked around 70,900 to 1 in mid-2025, meaning its training crawler pulled tens of thousands of pages for every visitor it sent. Traditional Googlebot sits closer to 5 to 1.
• Roughly 82% of AI bot activity is now training-oriented, up from about 72% a year earlier, so the bulk of AI crawling is the low-return kind.
• On the demand side, one platform reported that ChatGPT drove nearly 10% of new sign-ups by April 2026, up from under 1% six months earlier.

Put together, the picture is clear. Training crawlers consume enormous amounts of content and return almost nothing, while AI search is quickly becoming a real acquisition channel. The smart move is not to block everything or allow everything. It is to block the low-return training crawlers if you want, but always keep the search indexers open.

---

The Bots That Matter in 2026

To act on the split you need to know which user agent does which job. The table below groups the major crawlers by vendor and purpose.

A few clarifications that trip people up:

• GPTBot is not OAI-SearchBot. Blocking GPTBot opts you out of OpenAI training. It does not remove you from ChatGPT search, which relies on OAI-SearchBot and, in some cases, Bing's index.
• ClaudeBot is not Claude-SearchBot. The training bot and the search bot are separate strings, so a blanket "Claude" block can catch the wrong one.
• Google-Extended is a training opt-out only. It does not affect normal Googlebot crawling or your Google Search rankings. It governs whether your content trains Gemini.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a human asked the assistant to look at it. Blocking these can break real user requests, so treat them carefully.

The three jobs, in plain terms

1. Train — offline, batch collection to build the model's knowledge. Low direct payoff for publishers.
2. Index for search — builds the retrieval index that answer engines cite. High payoff, protect this.
3. Fetch on demand — grabs a page because a user pasted or referenced it live. Blocking hurts real users.

---

The Accidental Block Problem

Most AI crawler blocks are not deliberate. Nobody wakes up and decides to add Disallow: OAI-SearchBot to shut themselves out of ChatGPT. The blocks happen quietly, through defaults and blunt rules.

Three common causes:

• Hosting and CDN defaults. Some platforms began blocking AI crawlers by default for sites created after a certain date. If you never touched robots.txt, you may already be blocking search indexers without knowing it.
• Blanket "block all AI" snippets. Plugins and copy-pasted robots.txt blocks often list every AI user agent under one Disallow, catching search bots along with training bots.
• Legacy anti-scraping rules. Old rules written to stop content theft can match modern search crawlers whose names did not exist when the rule was written.

The scale is not trivial. A large share of news publishers were found to be blocking a search bot, often by accident, and paying for it in lost citations. Because the failure is invisible in normal analytics, it can persist for months.

The lesson: audit what you actually block, because the default may not be what you intended.

---

A Copy-Ready robots.txt Policy

Here is a balanced configuration for the most common goal: stay out of training corpora where you can, but remain fully citable in AI search. Adjust to taste.

# Allow AI search indexers (protect citations)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Googlebot
Allow: /

# Opt out of model training (optional)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

If you want maximum reach and do not mind training inclusion, simply allow everything and skip the training block. If you are a publisher who wants to keep content out of training for licensing reasons, the block above does that without sacrificing AI-search visibility.

What NOT to do

Avoid the single blanket rule that many templates still recommend:

# This is the mistake — it blocks search citation bots too
User-agent: *
Disallow: /

A wildcard disallow aimed at "AI bots" often catches OAI-SearchBot, PerplexityBot, and Bingbot. That is how brands accidentally erase themselves from AI answers.

---

robots.txt Is Not the Same as noindex

A subtle but important rule: disallowing a crawl is not the same as removing a page from results. They solve different problems, and combining them incorrectly causes trouble.

• Disallow in robots.txt tells a crawler not to fetch the page. But a page can still surface if it is discovered through links, because the crawler never read the page to see any instructions on it.
• noindex tells an engine not to include the page in results. For noindex to be seen, the crawler must be allowed to fetch the page in the first place.

The practical rule: never apply Disallow and noindex to the same URL at once. If you want a page kept out of results, allow the crawl and use noindex. If you block the crawl, the engine may never read your noindex and could still reference the URL.

For AI search specifically, the goal is usually the opposite: you want the search indexers to fetch and read your pages so they can cite them accurately. Blocking them guarantees the opposite outcome.

---

How to Audit Your Current AI Crawler Access

You do not need special tooling to find out whether you are accidentally blocking search bots. Three checks cover most cases.

1. Read your live robots.txt

Open https://yourdomain.com/robots.txt in a browser. Look for any Disallow: / rules tied to OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, or a catch-all User-agent: *. Any of those can suppress AI-search citations.

2. Check your server logs for search bots

Grep your access logs for the search indexer user agents:

grep -E "OAI-SearchBot|PerplexityBot|Claude-SearchBot|Bingbot" access.log

If you see training bots (GPTBot, ClaudeBot) but never the search indexers, a block or a crawl barrier may be keeping the citation crawlers out.

3. Confirm your CDN or host is not blocking by default

If your site is behind a CDN that shipped an AI-blocking default, robots.txt alone will not tell the whole story, because the block may happen at the edge before the request reaches your origin. Check your CDN's bot-management settings for an "AI crawlers" toggle and confirm the search indexers are allowed.

---

Beyond robots.txt: Being Citable Once You Are Crawlable

Opening the door to search crawlers is necessary but not sufficient. Once OAI-SearchBot and PerplexityBot can reach your pages, the content still has to be readable and quotable.

• Serve content in the initial HTML. Most AI crawlers do not execute JavaScript, so client-side-rendered content is invisible even to bots you have allowed. Render critical content server-side.
• Keep entity language consistent. Use one stable name for your brand, product, and category so retrieval systems can connect mentions into a single entity.
• Write self-contained passages. Answer engines often lift a single paragraph, so each one should carry a complete idea with an explicit subject.
• Add structured data server-side. Organization, Product, Article, and FAQ schema help machines interpret what a page is about.

Crawler access decides whether you are in the game. Content structure decides whether you get cited once you are.

---

FAQ: Training Bots vs Search Crawlers

If I block GPTBot, do I disappear from ChatGPT search?

No. GPTBot handles OpenAI training. ChatGPT search relies on OAI-SearchBot and, in some cases, Bing's index. Blocking GPTBot opts you out of training while leaving your ChatGPT-search eligibility intact, as long as OAI-SearchBot and Bingbot remain allowed.

What is the difference between ClaudeBot and Claude-SearchBot?

ClaudeBot is Anthropic's training and corpus crawler. Claude-SearchBot indexes pages for Claude's search-grounded answers. They are separate user agents, so you can block training while allowing search citation.

Does Google-Extended affect my Google rankings?

No. Google-Extended is a training opt-out for Gemini. It does not change how Googlebot crawls your site or how you rank in Google Search. It only governs whether your content is used to train Google's models.

Should I block all AI crawlers to protect my content?

Usually not. A blanket block also removes the search indexers that generate AI citations and referral traffic. If you have licensing concerns, block the training bots specifically and keep the search bots open.

Why do my pages still rank in Google but never appear in AI answers?

The most common cause is that a robots.txt rule or CDN default is blocking the AI-search indexers while Googlebot is still allowed. The pages remain in Google's index but never enter the AI retrieval layer, so they cannot be cited.

Are these bots guaranteed to obey robots.txt?

The major AI search crawlers generally respect robots.txt, and studies have found high compliance among the main bots. That is exactly why an accidental block is so costly: the crawlers obey it faithfully, including your mistakes.

---

Key Takeaways

• AI vendors run separate crawlers for training and for search. They are controlled by different user agents in robots.txt.
• Blocking all AI bots is now harmful. It removes you from AI-search citations, the fastest-growing referral channel of 2026.
• Block training bots if you want, but always allow the search indexers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Bingbot.
• Most blocks are accidental, caused by hosting defaults, blanket snippets, or legacy rules. Audit what you actually block.
• Disallow is not noindex. Never apply both to the same URL, and for AI search, let the indexers in.
• Access is only step one. Server-side rendering, consistent entities, and structured data decide whether you get cited once crawlable.

The training-versus-search split is one of the most consequential and least understood technical decisions in AI visibility today. It is also one of the easiest to get right: a few precise lines in robots.txt separate a brand that AI can cite from one it cannot even reach. Auditing your crawler access, and correcting an accidental block, is among the highest-leverage fixes available in a modern GEO program.

---

References

1. AI Crawler Access Control: robots & llms.txt Decision Matrix — digitalapplied.com
2. robots.txt for AI Crawlers: The 2026 Setup — okara.ai
3. Top 12 LLM Crawlers and What They Do (2026 Directory) — hyperleap.ai
4. AI Crawler Cheatsheet: User-Agents & Whether to Block — lawrencehitches.com
5. Should You Block AI Crawlers? A robots.txt Decision Guide — guptadeepak.com
6. AI bots robots.txt guide: GPTBot, ClaudeBot — soar.sh
7. Overview of OpenAI Crawlers — OpenAI Developers