AI Crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended
Nine user-agents cover most AI traffic: GPTBot, OAI-SearchBot, and ChatGPT-User for OpenAI; ClaudeBot and anthropic-ai for Anthropic; PerplexityBot for Perplexity; Google-Extended for Gemini; Applebot-Extended for Apple; and CCBot for Common Crawl. Allow them unless you have a specific reason not to: blocking does not affect Google rankings, but it does opt your site out of that ecosystem's understanding.
Key takeaways
- AI crawlers do three different jobs: training, index retrieval, and live browsing, decide per user-agent, not for all bots at once.
- Blocking is a clean opt-out from an ecosystem's understanding of you; it does not change how Google ranks your pages.
- Allow the major AI user-agents, keep abusive scrapers disallowed, and put a conservative
User-agent: *group last. - Verify with logs: watch for AI user-agent strings and confirm your robots.txt returns what you expect to crawler user-agents.
- The free audit checks AI crawler access as part of its roughly 40 checks.
The AI user-agents worth knowing
Every major AI provider publishes at least one crawler, and the jobs are not interchangeable. The distinction that matters is training versus retrieval versus live browsing: training crawlers collect data for model training, retrieval crawlers build the index an assistant queries, and browsing agents fetch a page live because a user asked for it right now.
Providers rename and re-role these agents periodically, so treat this list as current-but-provisional and check each provider's crawler documentation before you make blocking a policy. The pattern, though, is stable: one agent for training, one for retrieval, sometimes one for live fetch.
- GPTBot, OpenAI's crawler for training data. OAI-SearchBot, indexes pages for ChatGPT's search answers. ChatGPT-User, fetches a page live when a user asks ChatGPT to open it.
- ClaudeBot, Anthropic's training crawler. anthropic-ai, retrieves content for Claude's responses.
- PerplexityBot, builds the index behind Perplexity's cited answers.
- Google-Extended, the standalone control for Gemini apps' training and grounding; Google documents that it does not affect Search ranking.
- Applebot-Extended, opts your content out of Apple Intelligence training and related model improvement.
- CCBot, Common Crawl's crawler; its open dataset feeds many AI labs.
What blocking actually costs you
robots.txt blocking is a clean opt-out. A compliant crawler reads the rule, skips your pages, and your content stays out of that provider's training data and index. The cost is symmetric and easy to underestimate: if your site is blocked, an assistant drawing on that ecosystem cannot describe your pricing, recommend your service, or cite your pages, it will answer from competitors who left the door open.
Allowing the crawlers is equally safe for classic search. Per Google's documentation on AI features and Search, Google-Extended is a separate control that does not affect your search ranking either way, and the other AI agents are not search crawlers at all. So the decision is about visibility inside AI products, not about SEO points. Firms that sell expertise usually want to be understood by as many ecosystems as possible; publishers fighting content scraping may block selectively. Both positions are defensible, the failure mode is deciding by default instead of on purpose.
A copy-ready robots.txt that allows AI crawlers
This snippet allows the major AI crawlers while keeping abusive scrapers out. Structure matters: one shared allow group, then your deny rules for known bad actors, then a conservative wildcard group last.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
Disallow: /
User-agent: *
Allow: /
Disallow: /admin
Disallow: /account
Merge it into your existing file rather than replacing it, keep your current security, staging, and parameter rules. Consecutive User-agent lines share the directives that follow them, which is what keeps this file readable. Then test it: request /robots.txt with a crawler user-agent and confirm the rules you expect come back.
How to verify with logs
Rules on paper are not verification. These checks take about ten minutes:
None of these checks predicts citations or traffic, they confirm access, which is only the first step of the pipeline. What happens after retrieval depends on comprehension and clarity further down the line.
- Fetch as the bot. Request
/robots.txtwith aUser-Agent: GPTBotheader (curl works) and read what comes back, then repeat for one or two other agents. - Watch server or CDN logs for hits carrying the AI user-agent strings. A
GPTBotrequest in the log proves the allow rule works, and a request while it is disallowed tells you the crawler is ignoring robots.txt, which is worth knowing. - Check referral sources in analytics for chatgpt.com, perplexity.ai, or claude.ai. Referrals are a downstream signal that retrieval happened, not a guarantee of future traffic.
- Re-test after every robots.txt edit. A wildcard group in the wrong place silently re-blocks everything above it.
What the audit checks
The free audit checks exactly this: whether AI user-agents are allowed or blocked, whether robots.txt returns correctly to crawler user-agents, and how that interacts with your sitemap and schema. Scoring is Discovery 40% / Data Quality 40% / Actionability 20% across roughly 40 checks, and the output is a prioritized fix list with copy-ready code, robots.txt snippets included.
If you want the reasoning behind the checks first, start at the guides hub; if you would rather talk it through, book a 30-minute call.
Frequently asked questions
Should I block GPTBot on my website?
Do AI crawlers affect my SEO rankings?
How do I know if an AI bot is visiting my site?
What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
Put this guide to work on your site.
Run the free audit for an instant score and prioritized fix list, or book a 30-minute call and we scope the work for you.
We do not guarantee AI citations, rankings, traffic, leads, revenue, or purchases.
Related guides
Foundations · 6 min read
llms.txt: How to Create One (With a Template)
llms.txt is a curated markdown file at your site root for AI clients. What it is, when it helps, a copy-ready template, and honest limits on its impact.
Foundations · 7 min read
How AI Assistants See Your Website (From Crawl to Citation)
The four steps between your site and an AI answer, crawler access, retrieval, comprehension, and citation, and what degrades each one.
Playbooks · 8 min read
How to Audit Your Website for AI Readiness (Free Method)
Measure AI readiness with a weighted audit: Discovery 40%, Data Quality 40%, Actionability 20%. Manual steps, plus the free 30-second audit with a fix list.