robots.txt for AI Crawlers: The 2026 Cheat Sheet (GPTBot, ClaudeBot, PerplexityBot…)

Most AI companies now run separate bots for training, search and live user requests. Block the wrong one and you vanish from AI answers. Here's every bot, and a robots.txt you can copy.

The three kinds of AI bot

Training crawlers collect pages to train future models. Blocking them doesn't affect whether you appear in answers today.
Search crawlers index pages so the assistant can cite them. Block these and you disappear from AI answers.
User fetchers load a page live because someone asked about it. Some vendors say robots.txt may not apply to these.

Every bot, at a glance

Company	Training	Search / index	User fetch
OpenAI	`GPTBot`	`OAI-SearchBot`	`ChatGPT-User`robots.txt "may not apply"
Anthropic	`ClaudeBot`	`Claude-SearchBot`	`Claude-User`all three honour robots.txt
Perplexity	none (says it doesn't train foundation models)	`PerplexityBot`	`Perplexity-User`generally ignores robots.txt
Google	`Google-Extended`control token for Gemini training and grounding	`Googlebot`also powers AI Overviews / AI Mode	—
Apple	`Applebot-Extended`usage signal, doesn't crawl	`Applebot`	—
Microsoft	—	`Bingbot`Copilot uses the Bing index	—
Meta	`meta-externalagent`	`meta-webindexer`	`meta-externalfetcher`may bypass robots.txt
Amazon	`Amazonbot`	`Amzn-SearchBot`	`Amzn-User`
Mistral	`MistralAI-Training`	`MistralAI-Index`	`MistralAI-User`
DuckDuckGo	—	—	`DuckAssistBot`not used for training
Common Crawl	`CCBot`dataset widely used for training	—	—

ByteDance's Bytespider has no official documentation and has been reported to ignore robots.txt, so also block it at your firewall if you care. xAI (Grok) doesn't document a crawler at all.

The copy-paste robots.txt

This keeps you visible in AI search and answers, and opts you out of model training. Delete the "optional" block if you're happy to be used for training.

# --- AI search / answer engines: allow (needed to be cited) ---
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: Amzn-SearchBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
Allow: /

# --- Fetches triggered by a user's question: allow ---
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
Allow: /

# --- Classic search (also feeds AI Overviews, AI Mode, Copilot) ---
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /

# --- OPTIONAL: opt out of model training (delete to allow) ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Listing several User-agent lines above one rule set is valid under RFC 9309. Changes usually take up to 24 hours to apply (OpenAI and Perplexity both say so; DuckDuckGo says 72 hours).

Four gotchas

1. Google-Extended doesn't control AI Overviews

Google says Google-Extended "does not impact a site's inclusion in Google Search". AI Overviews and AI Mode are part of Search, so they follow Googlebot and snippet controls like nosnippet and max-snippet. Those also affect normal search results.

2. Bing and Copilot use meta tags, not a separate bot

NOCACHE limits Copilot to your URL, title and snippet. NOARCHIVE keeps the page out of Copilot answers entirely. If you want to be cited, use neither.

3. Your CDN can silently override robots.txt

Cloudflare made blocking AI crawlers the default for new domains in July 2025. In 2026 it grouped bots into Search, Agent and Training, and warned that multi-purpose crawlers fall under the strictest rule, so blocking "Training" can block Googlebot, Applebot and Bingbot too. Check your CDN settings, not just the file.

4. llms.txt is a proposal, not a standard

llms.txt (proposed by Jeremy Howard in 2024) is a Markdown summary of your site for language models. Google says you don't need "AI text files" and that Search ignores them, and no major AI search engine has said it uses them. It's harmless to add, and Chrome's Lighthouse has an experimental check for it, but don't expect it to change whether you're cited.

How to check yours in 30 seconds

Open yoursite.com/robots.txt and search for each search bot above.
Look for a blanket User-agent: * / Disallow: / on staging-copied configs.
Check your CDN or WAF for "AI bot" or "AI scrapers" toggles.
Or run npx getplugd audit yoursite.com, which checks all of it.

robots.txt for AI crawlers: the 2026 cheat sheet