# Dex robots.txt — AI-crawler-aware. # Humans: see https://getdex.com/llms.txt for an AI-friendly site map. # ---------------------------------------------------------------------- # Default tier — traditional search crawlers. Full access. # Content-Signal (Cloudflare Content Signals Policy): public site # content may be indexed for search and used as AI-assistant input, # but should NOT be used to train AI models without a license. # ---------------------------------------------------------------------- User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Disallow: /app Disallow: /app/ Disallow: /appv3 Disallow: /appv3/ # ---------------------------------------------------------------------- # AI crawler tier 1 — user-agent crawlers (fetch on user request, do not # train on content). Full access so agents can answer user questions in # real time. # ---------------------------------------------------------------------- User-agent: ChatGPT-User Allow: / Disallow: /app Disallow: /appv3 User-agent: OAI-SearchBot Allow: / Disallow: /app Disallow: /appv3 User-agent: PerplexityBot Allow: / Disallow: /app Disallow: /appv3 User-agent: Perplexity-User Allow: / Disallow: /app Disallow: /appv3 User-agent: Claude-User Allow: / Disallow: /app Disallow: /appv3 User-agent: Claude-SearchBot Allow: / Disallow: /app Disallow: /appv3 User-agent: DuckAssistBot Allow: / Disallow: /app Disallow: /appv3 User-agent: YouBot Allow: / Disallow: /app Disallow: /appv3 User-agent: Applebot-Extended Allow: / Disallow: /app Disallow: /appv3 # ---------------------------------------------------------------------- # AI crawler tier 2 — training crawlers (index content for model # training). Allowed on public marketing + documentation content. # ---------------------------------------------------------------------- User-agent: GPTBot Allow: / Disallow: /app Disallow: /appv3 User-agent: ClaudeBot Allow: / Disallow: /app Disallow: /appv3 User-agent: anthropic-ai Allow: / Disallow: /app Disallow: /appv3 User-agent: Google-Extended Allow: / Disallow: /app Disallow: /appv3 # CCBot (Common Crawl) and Bytespider are pure training crawlers with no # user-assistant surface. Hard-block them — the Content-Signal above says # ai-train=no and these are the two UAs that most obviously violate that. User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Allow: / Disallow: /app Disallow: /appv3 User-agent: Meta-ExternalAgent Allow: / Disallow: /app Disallow: /appv3 User-agent: DeepSeekBot Allow: / Disallow: /app Disallow: /appv3 User-agent: MistralAI-User Allow: / Disallow: /app Disallow: /appv3 User-agent: Cohere-AI Allow: / Disallow: /app Disallow: /appv3 # ---------------------------------------------------------------------- # Sitemaps # ---------------------------------------------------------------------- Sitemap: https://getdex.com/sitemap-index.xml Sitemap: https://getdex.com/sitemap-0.xml Sitemap: https://getdex.com/blog/sitemap.xml