The 2026 AI Crawler & Robots.txt Specification Guide
A common misconception among web developers is that all AI bots are training scrapers. In reality, modern search bots like OAI-SearchBot and Claude-SearchBot exist solely to deliver citations and referrals to your website. Blocking them in robots.txt or Cloudflare WAF causes instant invisibility across generative search.
Critical Difference: Search Bots vs. Training Bots
Search Bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) perform live web lookups to cite your practice or business when users ask questions. Training Bots (GPTBot, ClaudeBot) ingest content for foundational model pre-training. A proper AEO configuration separates these two roles, keeping search channels wide open.
Recommended 2026 AI Robots.txt Block
# Frontier AI Answer Engines (Explicitly Permitted) User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: bingbot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Applebot-Extended Allow: / # Canonical endpoints Sitemap: https://cernitur.com/sitemap.xml
The Cloudflare Edge WAF Trap
Even with a fully permitted robots.txt, Cloudflare's default Bot Fight Mode or Managed Challenge rules often block verified AI crawlers with HTTP 403 responses.
To fix this, create a custom WAF Skip Rule matching:
Verify Your Site's Crawler Accessibility
Run Cernitur's free 100-point diagnostic to check if your domain accidentally blocks OpenAI, Perplexity, or Claude.