Guides · HolicAudit
Should you block GPTBot in robots.txt?
Published 2 November 2026
In short: blocking training crawlers like GPTBot keeps your content out of future model training but doesn't remove you from AI search. Blocking search crawlers like OAI-SearchBot or PerplexityBot does. Decide separately for each group, then check your CDN isn't blocking them for you.
Two kinds of AI crawler
| User agent | Company | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results and citations |
| ChatGPT-User | OpenAI | Pages ChatGPT opens because a user asked; OpenAI says robots.txt may not apply |
| Claude-SearchBot | Anthropic | Claude search results |
| PerplexityBot | Perplexity | Perplexity's search index |
| GPTBot | OpenAI | Training OpenAI models |
| ClaudeBot | Anthropic | Training Anthropic models |
| Google-Extended | A control token for Gemini training and grounding; no separate crawler, and it doesn't affect Google Search | |
| Applebot-Extended | Apple | A control token for Apple model training; doesn't affect Apple search features |
| CCBot | Common Crawl | A public web archive that many AI models are trained on |
| Bytespider | ByteDance | Training ByteDance models |
OpenAI states that its settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Google AI Overviews use regular Googlebot, so Google-Extended doesn't remove you from them.
Reasons to block training crawlers
- Your content is the product: paid articles, courses, original research, stock media.
- You don't want your work used to train models without a licence.
- Crawler traffic costs you real bandwidth.
Reasons to allow them
- You sell a product or service and want models to know it exists, even when they answer without browsing.
- Your content is marketing: the more places it shows up, the better.
Copy-paste rules
Allow AI search, block training (a common middle ground):
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlRemember how groups work: a crawler that matches a named group ignores the * group entirely. If your * group disallows /admin/, repeat that line in every named group too.
robots.txt is a request, not a wall
The major AI companies say their crawlers respect robots.txt, but it's voluntary and doesn't stop scrapers that ignore it. For real control, use your CDN's bot management or firewall rules. Many CDNs now offer one-click AI crawler blocking, which works the other way too: if you want AI search traffic, make sure that setting isn't quietly blocking the search crawlers you just allowed.
Check what your site allows
Our AI visibility check reads your robots.txt the way crawlers do (most specific group, longest matching rule) and lists every crawler above as allowed or blocked for the page you enter. For general robots.txt testing, use the robots.txt checker.
Free tool: AI visibility check
Can ChatGPT, Perplexity, Claude and Google AI Overviews read and cite your page? Checks AI crawler rules, llms.txt, server-rendered text and structured data.
Frequently asked questions
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's crawler and has nothing to do with Google Search. Google-Extended doesn't affect Google Search rankings either.
If I block GPTBot, will ChatGPT stop mentioning my site?
Not from search. ChatGPT search uses OAI-SearchBot, which is a separate setting. Answers that come from the model's built-in knowledge may know less about you over time.
How quickly do changes take effect?
OpenAI says ChatGPT search can take about 24 hours to reflect a robots.txt change. Google usually caches robots.txt for up to a day.
Do all AI crawlers obey robots.txt?
The big named crawlers say they do. Unknown scrapers may not, so use firewall or CDN bot rules if you need enforcement.