Quick answer: AI companies use separate crawlers for different jobs. Some collect content to train AI models (GPTBot, ClaudeBot), some build the index behind AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and some fetch a page only when a user asks about it (ChatGPT-User, Claude-User, Perplexity-User). You control each one separately in robots.txt. For most publishers who want to be cited in AI answers, the sensible setup is to allow the search crawlers and decide on the training crawlers as a separate business choice.
Blocking “AI bots” with one blanket rule is one of the most common mistakes site owners make right now. It can quietly remove you from ChatGPT search and Perplexity results, even if all you wanted was to keep your content out of model training. This guide lists the crawlers that matter in 2026, what each one does according to its owner’s documentation, and the exact robots.txt lines to allow or block them.
Training crawlers vs search crawlers vs user agents
Before editing robots.txt, it helps to know the three kinds of AI bots you are dealing with:
- Training crawlers gather public web pages that may be used to train generative AI models. Blocking them is a signal that your content should not go into future training data. It does not usually affect whether you appear in AI search answers.
- Search crawlers crawl and index pages so an AI search product can find, quote and link to them. Block these and you can disappear from that product’s answers.
- User-triggered agents fetch a specific page because a person asked the assistant about it or pasted a link. They are not general crawlers, and some companies say robots.txt may not apply to them.
Google and Apple add a twist: they do not run separate AI training crawlers. Instead they offer a robots.txt token (Google-Extended, Applebot-Extended) that controls how content already crawled by their normal bots may be used for AI.
The main AI crawlers in 2026
| User agent | Company | Job | Follows robots.txt? | If you block it |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes | Signals your content should not be used to train OpenAI’s foundation models |
| OAI-SearchBot | OpenAI | Search | Yes | Your pages can drop out of ChatGPT search results (they may still appear as plain navigational links) |
| ChatGPT-User | OpenAI | User-triggered | May not apply | Limited effect; OpenAI says robots.txt rules may not apply to user-initiated visits |
| ClaudeBot | Anthropic | Training | Yes | Signals your future content should be excluded from Anthropic’s training data |
| Claude-SearchBot | Anthropic | Search | Yes | Your pages may be less visible in Claude’s search answers |
| Claude-User | Anthropic | User-triggered | Yes | Claude can’t fetch your page when a user asks about it |
| PerplexityBot | Perplexity | Search | Yes (Perplexity recommends allowing it) | Your site is less likely to be surfaced and linked in Perplexity answers |
| Perplexity-User | Perplexity | User-triggered | Generally no | Little effect; Perplexity says it generally ignores robots.txt |
| Google-Extended | AI-use token (not a crawler) | Yes | Content can’t be used to improve Gemini Apps and Vertex AI models; Google Search is unaffected | |
| Applebot-Extended | Apple | AI-use token (not a crawler) | Yes | Content can’t be used to train Apple’s foundation models; Apple search results are unaffected |
Sources for each row are listed at the end of this article. Companies add and rename crawlers from time to time, so check their pages before making a big change.
OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI is explicit that its settings are independent. Disallowing GPTBot tells OpenAI not to use your content for training its foundation models, while OAI-SearchBot decides whether your pages can surface in ChatGPT search. OpenAI recommends allowing OAI-SearchBot if you want to appear in search results and says robots.txt changes can take about 24 hours to show up there.
ChatGPT-User is different. It visits a page when someone using ChatGPT or a custom GPT asks for it. OpenAI says it is not used for automatic crawling, robots.txt rules may not apply to it, and it does not decide whether you appear in search. If you allow both GPTBot and OAI-SearchBot, OpenAI notes it may use one crawl for both purposes. For more on getting cited, see how to rank in ChatGPT search.
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic runs three bots. ClaudeBot collects content that may be used for training; blocking it signals that your future content should be excluded from training data. Claude-SearchBot crawls to improve Claude’s search results, and Claude-User fetches pages when a Claude user asks a question. Anthropic says blocking either of the last two may reduce your visibility in Claude’s answers.
Anthropic states that its bots honour robots.txt and support the non-standard Crawl-delay directive if you want to slow them down. It also advises against IP blocking, because that stops its bots from reading your robots.txt and so may not work as a reliable opt-out.
Perplexity: PerplexityBot and Perplexity-User
PerplexityBot surfaces and links websites in Perplexity’s answers, and Perplexity says it is not used to crawl content for training foundation models. Perplexity-User fetches a page when a user asks a question and, in Perplexity’s words, generally ignores robots.txt because a person triggered the request. Changes can take up to 24 hours to apply. Our Perplexity AI SEO guide covers the rest of the picture.
Google-Extended and Applebot-Extended
Google-Extended is a product token, not a separate crawler. Google’s regular crawler fetches your pages; the token controls whether that content can be used to improve Gemini Apps and Vertex AI generative APIs. Google says it does not affect inclusion or ranking in Google Search. Because AI Overviews and AI Mode are part of Search, blocking Google-Extended does not remove you from them; those features follow Search controls such as noindex and nosnippet. See our Google AI Mode SEO guide for how that works.
Applebot-Extended works the same way for Apple. Apple says it does not crawl pages itself, that disallowing it opts your content out of training Apple’s foundation models, and that pages which disallow it can still appear in Apple’s search results.
Three robots.txt setups you can copy
Each block below starts with User-agent: for one bot, followed by its rule. A bot follows the most specific group that names it, so list AI crawlers by name rather than relying only on User-agent: *.
Setup 1: Appear in AI search, stay out of AI training
This is the most common choice for publishers who want citations and traffic from AI answers but don’t want their content used to train models.
# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Block AI training crawlers and tokens
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Setup 2: Allow everything (maximum AI visibility)
If you want your content available for both AI search and training, you don’t need any AI-specific lines. A standard file that allows all crawlers is enough:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap_index.xmlSetup 3: Block all AI crawlers you can control
Use this only if you are sure you don’t want to appear in AI search answers. Remember that user-triggered agents such as ChatGPT-User and Perplexity-User may still fetch pages that a user asks about.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /You can also block a single folder instead of the whole site, for example Disallow: /members/, to keep paid or private sections out while leaving public articles open.
How to edit robots.txt in WordPress
- Check what you have now. Open
yourdomain.com/robots.txtin a browser. Many WordPress sites serve a virtual file generated by WordPress or an SEO plugin. - Edit it through your SEO plugin. In Yoast SEO, go to Yoast SEO → Tools → File editor. Rank Math and other SEO plugins have a similar robots.txt editor. If there is no editor, upload a
robots.txtfile to your site’s root folder by FTP or your host’s file manager. - Keep your existing rules. Add the AI groups below what is already there, and keep your
Sitemap:line. - Clear your cache if you use a caching plugin or CDN, so crawlers see the new file.
- Test it. Paste your URL into our free Robots.txt Tester and check each user agent against a few important pages. You can also build a clean file from scratch with the Robots.txt Generator.
Common mistakes to avoid
- Blocking everything AI with one rule. Copy-pasted “block AI” lists often include search crawlers like OAI-SearchBot and PerplexityBot, which can cut you out of AI answers.
- Thinking Google-Extended controls AI Overviews. It doesn’t. It only covers Gemini Apps and Vertex AI use. AI Overviews and AI Mode depend on normal Search indexing.
- Blocking Googlebot to stop AI. This removes you from Google Search entirely, which is almost never what you want.
- Forgetting your firewall or CDN. Some hosts, CDNs and security plugins have their own “block AI bots” settings. If those block a crawler, your robots.txt never gets a say. Check those settings if AI search tools can’t seem to reach your site.
- Expecting instant results. OpenAI and Perplexity both say changes can take up to about 24 hours to take effect, and content that was already collected isn’t removed retroactively.
What robots.txt can and can’t do
robots.txt is a public set of instructions, not a lock. Reputable AI companies say their crawlers follow it, but it doesn’t stop a bot that chooses to ignore it, and user-triggered agents may not apply it. To keep content truly private, put it behind a login. To stop a page appearing in Google’s AI features while staying in Search, use snippet controls like nosnippet or data-nosnippet on the relevant text.
If you want to confirm a visit really came from a given company, OpenAI, Anthropic and Perplexity publish the IP ranges their bots use. Matching both the user agent and the IP range is more reliable than the user agent alone, since anyone can fake a user-agent string.
Which setup should you choose?
If AI search traffic matters to your business, start with Setup 1 or Setup 2. Being cited in ChatGPT, Claude and Perplexity answers is a growing source of visits, as our AI search statistics show, and blocking the search crawlers gives that up. Whether to also allow training crawlers is a judgement call about how you want your content used; it has no documented effect on whether you appear in AI answers.
Once your crawler settings are right, the next step is making your content easy for AI tools to understand and quote. Our guides to optimising your website for AI search and generative engine optimisation cover that in detail.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot controls training use. ChatGPT search relies on OAI-SearchBot, which you control separately. If you block only GPTBot, your pages can still appear in ChatGPT search.
Does blocking Google-Extended affect my Google rankings?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It also doesn’t keep you out of AI Overviews or AI Mode.
Will blocking AI crawlers remove content they already collected?
Not according to the companies’ documentation. Anthropic, for example, says blocking ClaudeBot signals that your future content should be excluded. Content collected earlier isn’t covered.
Can I block AI crawlers from only part of my site?
Yes. Use a Disallow rule with a folder path, such as Disallow: /premium/, under the crawler’s user agent.
Why does an AI tool still visit my page after I blocked it?
It may be a user-triggered agent like ChatGPT-User or Perplexity-User, which may not follow robots.txt, or the change may not have taken effect yet. Check the user agent and IP in your server logs.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may access. llms.txt is a separate, proposed file meant to guide AI tools to your most useful content, and support for it varies.
Sources
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google Search Central: Google’s common crawlers (Google-Extended)
- Apple: About Applebot (Applebot-Extended)
