# AI Architectures — robots.txt (AI crawl policy updated 2026-10-08) # Search engines and AI search/assistant bots are welcome: they send buyers and citations. # Crawlers of companies that run an assistant (OpenAI, Anthropic, Google, Apple, Amazon, Meta, Cohere) are allowed. # Pure training-data scrapers without an assistant (Common Crawl, ByteDance, data resellers) are not. # A crawler obeys only its most specific group, so every group repeats the private routes. # === Everyone else === # default User-agent: * Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # === Search engines === # Google Search + AI Overviews User-agent: Googlebot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Google URL Inspection / rich results User-agent: Google-InspectionTool Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Bing; Bing's index grounds Microsoft Copilot and ChatGPT search User-agent: Bingbot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Siri, Spotlight, Apple Intelligence answers User-agent: Applebot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # DuckDuckGo User-agent: DuckDuckBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Yandex User-agent: YandexBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Brave Search User-agent: Bravebot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # You.com User-agent: YouBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # === AI search and assistant bots (live answers, citations) === # ChatGPT search index (citations in ChatGPT) User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # ChatGPT fetching a page for a user User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Claude search index (citations in Claude) User-agent: Claude-SearchBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Claude fetching a page for a user User-agent: Claude-User Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # legacy Anthropic fetch token User-agent: Claude-Web Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Perplexity search index User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Perplexity fetching a page for a user User-agent: Perplexity-User Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # DuckDuckGo AI answers User-agent: DuckAssistBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Mistral Le Chat fetches User-agent: MistralAI-User Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Meta AI fetching a link for a user User-agent: Meta-ExternalFetcher Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Alexa+/Rufus search User-agent: Amzn-SearchBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Alexa+ fetching a page for a user User-agent: Amzn-User Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # === AI model crawlers of assistant vendors: allowed === # OpenAI models (ChatGPT knows us) User-agent: GPTBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Anthropic models (Claude knows us) User-agent: ClaudeBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # legacy Anthropic token User-agent: anthropic-ai Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Gemini training AND Gemini app grounding User-agent: Google-Extended Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Apple Intelligence models User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Alexa / Amazon AI User-agent: Amazonbot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Meta AI User-agent: Meta-ExternalAgent Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Cohere User-agent: cohere-ai Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # Common Crawl: allowed again 2026-10-09 (owner) — its corpus feeds many AI models User-agent: CCBot Allow: / Disallow: /admin Disallow: /oauth-success Disallow: /reset-password Disallow: /rewarded Disallow: /api/ # === Training-data scrapers without an assistant: blocked === # ByteDance: ignores crawl budgets User-agent: Bytespider Disallow: / # data resale crawler User-agent: Diffbot Disallow: / # bulk image scraper User-agent: img2dataset Disallow: / # data resale crawler User-agent: Omgilibot Disallow: / # training-data crawler User-agent: Timpibot Disallow: / # === Site information === Sitemap: https://ai-architectures.com/sitemap.xml # For AI assistants: https://ai-architectures.com/llms.txt (summary) and https://ai-architectures.com/llms-full.txt (full reference)