# ============================================================ # robots.txt — ladym.com # Lady M Cake Boutique # Last updated: April 2026 # # Strategy: MAXIMUM AI VISIBILITY # Lady M welcomes indexing by all major search engines and AI # answer engines to support brand discovery and citation. # Sensitive transactional and account paths are restricted for # all crawlers to protect user privacy and server resources. # # For questions: hello@ladym.com # ============================================================ # ────────────────────────────────────────────────────────────── # SECTION 1 — GOOGLE (Search + AI) # ────────────────────────────────────────────────────────────── # Googlebot: core web search indexer User-agent: Googlebot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /search Disallow: /cdn-cgi/ Disallow: /s/ # Google-Extended: Gemini / AI Overviews training # Allowing ensures Lady M content surfaces in Google AI answers User-agent: Google-Extended Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /search Disallow: /cdn-cgi/ Disallow: /s/ # Googlebot-Image User-agent: Googlebot-Image Allow: / Disallow: /cdn-cgi/ # Storebot-Google: Google Shopping crawler User-agent: Storebot-Google Allow: / # ────────────────────────────────────────────────────────────── # SECTION 2 — MICROSOFT (Bing Search + Copilot) # ────────────────────────────────────────────────────────────── User-agent: Bingbot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /search Disallow: /cdn-cgi/ Disallow: /s/ # MSNBot: legacy Bing crawler — keep for compatibility User-agent: MSNBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account # ────────────────────────────────────────────────────────────── # SECTION 3 — OPENAI (ChatGPT Search + Training) # ────────────────────────────────────────────────────────────── # GPTBot: OpenAI training data crawler # Allowing ensures Lady M facts are built into GPT model knowledge User-agent: GPTBot Allow: /collections/ Allow: /items/ Allow: /who-we-are Allow: /boutiques Allow: /blog/ Allow: /collections/cakes/about Allow: /rewards Allow: /faqs Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /search Disallow: /cdn-cgi/ Disallow: /s/ # OAI-SearchBot: ChatGPT real-time web search retrieval # This drives citations in ChatGPT answers — allow broadly User-agent: OAI-SearchBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ # ChatGPT-User: user-triggered browsing within ChatGPT User-agent: ChatGPT-User Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ # ────────────────────────────────────────────────────────────── # SECTION 4 — ANTHROPIC (Claude) # ────────────────────────────────────────────────────────────── # ClaudeBot: Anthropic's primary web crawler User-agent: ClaudeBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ Disallow: /s/ # Claude-SearchBot: Claude AI search retrieval User-agent: Claude-SearchBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # Claude-User: user-initiated Claude browsing sessions User-agent: Claude-User Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # Legacy Anthropic agents (deprecated July 2024 — retained for safety) User-agent: anthropic-ai Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # ────────────────────────────────────────────────────────────── # SECTION 5 — PERPLEXITY AI # ────────────────────────────────────────────────────────────── # PerplexityBot: Perplexity.ai search index builder User-agent: PerplexityBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ # Perplexity-User: real-time user-triggered Perplexity retrieval User-agent: Perplexity-User Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # ────────────────────────────────────────────────────────────── # SECTION 6 — APPLE (Siri + Spotlight + Apple Intelligence) # ────────────────────────────────────────────────────────────── User-agent: Applebot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ # Applebot-Extended: Apple Intelligence AI training User-agent: Applebot-Extended Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # ────────────────────────────────────────────────────────────── # SECTION 7 — META (Meta AI / Llama) # ────────────────────────────────────────────────────────────── User-agent: FacebookBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password User-agent: meta-externalagent Allow: /collections/ Allow: /items/ Allow: /who-we-are Allow: /boutiques Allow: /blog/ Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ Disallow: /s/ # ────────────────────────────────────────────────────────────── # SECTION 8 — AMAZON (Alexa / AWS AI) # ────────────────────────────────────────────────────────────── User-agent: Amazonbot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /cdn-cgi/ # ────────────────────────────────────────────────────────────── # SECTION 9 — DUCKDUCKGO # ────────────────────────────────────────────────────────────── User-agent: DuckAssistBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password User-agent: DuckDuckBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # ────────────────────────────────────────────────────────────── # SECTION 10 — OTHER TRUSTED CRAWLERS # ────────────────────────────────────────────────────────────── # YouBot (You.com AI search) User-agent: YouBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # cohere-ai (Cohere AI) User-agent: cohere-ai Allow: /collections/ Allow: /items/ Allow: /who-we-are Allow: /boutiques Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /s/ # CCBot (Common Crawl — feeds many open LLMs) User-agent: CCBot Allow: /collections/ Allow: /items/ Allow: /who-we-are Allow: /boutiques Allow: /blog/ Allow: /faqs Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /s/ # Slurp (Yahoo Search) User-agent: Slurp Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # Bytespider (ByteDance / TikTok) # Restricted to public marketing content only — known aggressive crawler User-agent: Bytespider Allow: /collections/ Allow: /items/ Allow: /who-we-are Allow: /boutiques Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /s/ Disallow: /search # ────────────────────────────────────────────────────────────── # SECTION 11 — SEO TOOLS (Allow read-only for auditing) # ────────────────────────────────────────────────────────────── # Semrush User-agent: SemrushBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # Ahrefs User-agent: AhrefsBot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # Moz User-agent: rogerbot Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password # ────────────────────────────────────────────────────────────── # SECTION 12 — BLOCKED CRAWLERS # ────────────────────────────────────────────────────────────── # Aggressive scrapers, spam harvesters, and bad actors that # provide no value to Lady M's SEO or AI visibility goals. # ────────────────────────────────────────────────────────────── User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SiteExplorer Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: PetalBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Exabot Disallow: / User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / # ────────────────────────────────────────────────────────────── # SECTION 13 — DEFAULT (all other crawlers) # ────────────────────────────────────────────────────────────── User-agent: * Allow: / Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /password Disallow: /search Disallow: /cdn-cgi/ Disallow: /s/ Disallow: /*.json$ # ────────────────────────────────────────────────────────────── # SITEMAPS # ────────────────────────────────────────────────────────────── Sitemap: https://www.ladym.com/sitemap.xml