===================================================================
Optimized Production Robots.txt - eCommerceFastlane.com
Verified for 2026 Standards - AI Visibility & SEO Efficiency
Updated: March 14, 2026
Change: Split Group 1 into 1A (Search Engines) and 1B (AI Bots)
Reason: AI bots only need English content for citations.
Blocking translated locales from AI crawlers saves significant
origin bandwidth without reducing citation potential.
===================================================================

-------------------------------------------------------------------
GROUP 1A: SEARCH ENGINES (Full Access Including Translations)
-------------------------------------------------------------------
Search engines need full access to ALL language versions for
international search indexing. Google, Bing, Yandex, and social
crawlers retain complete access to every locale.
-------------------------------------------------------------------

User-agent: Googlebot
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Bingbot
User-agent: YandexBot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: facebookexternalhit
User-agent: meta-externalagent
User-agent: Meta-ExternalFetcher
User-agent: Bytespider
Allow: /
Allow: /wp-json/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /readme.html
Disallow: /license.txt
# Duplicate Content & Efficiency Blocks (CRITICAL)
Disallow: /?s=
Disallow: /*?s=
Disallow: /*?replytocom
Disallow: /trackback/
Disallow: /*?utm_
Disallow: /*?share=
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /author/
Disallow: /20*/
Disallow: /page/*/

-------------------------------------------------------------------
GROUP 1B: AI CITATION BOTS (English Only)
-------------------------------------------------------------------
AI models build citation indices from English content.
Translated versions add zero citation value but multiply
origin bandwidth by 45x per page crawled.
Full English content access preserved for maximum AI visibility.
Translated locale prefixes blocked to reduce origin load.
-------------------------------------------------------------------

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ChatGPT-User/2.0
User-agent: ClaudeBot
User-agent: claude-web
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amazonbot
User-agent: YouBot
User-agent: PhindBot
User-agent: ExaBot
User-agent: AndiBot
User-agent: DuckAssistBot
User-agent: cohere-ai
User-agent: omgili
User-agent: omgilibot
Allow: /
Allow: /wp-json/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /readme.html
Disallow: /license.txt
# Duplicate Content & Efficiency Blocks (CRITICAL)
Disallow: /?s=
Disallow: /*?s=
Disallow: /*?replytocom
Disallow: /trackback/
Disallow: /*?utm_
Disallow: /*?share=
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /author/
Disallow: /20*/
Disallow: /page/*/
# Block Translated Locale Prefixes (Bandwidth Protection)
# AI bots only need English content for citation indexing.
# All 45 GTranslate language subdirectories listed below.
Disallow: /af/
Disallow: /sq/
Disallow: /am/
Disallow: /ar/
Disallow: /hy/
Disallow: /az/
Disallow: /eu/
Disallow: /be/
Disallow: /bn/
Disallow: /bs/
Disallow: /bg/
Disallow: /ca/
Disallow: /ceb/
Disallow: /zh-CN/
Disallow: /zh-TW/
Disallow: /co/
Disallow: /hr/
Disallow: /cs/
Disallow: /da/
Disallow: /nl/
Disallow: /eo/
Disallow: /et/
Disallow: /fi/
Disallow: /fr/
Disallow: /fy/
Disallow: /gl/
Disallow: /ka/
Disallow: /de/
Disallow: /el/
Disallow: /gu/
Disallow: /ht/
Disallow: /ha/
Disallow: /haw/
Disallow: /he/
Disallow: /hi/
Disallow: /hmn/
Disallow: /hu/
Disallow: /is/
Disallow: /ig/
Disallow: /id/
Disallow: /ga/
Disallow: /it/
Disallow: /ja/
Disallow: /jw/
Disallow: /kn/
Disallow: /kk/
Disallow: /km/
Disallow: /rw/
Disallow: /ko/
Disallow: /ku/
Disallow: /ky/
Disallow: /lo/
Disallow: /la/
Disallow: /lv/
Disallow: /lt/
Disallow: /lb/
Disallow: /mk/
Disallow: /mg/
Disallow: /ms/
Disallow: /ml/
Disallow: /mt/
Disallow: /mi/
Disallow: /mr/
Disallow: /mn/
Disallow: /my/
Disallow: /ne/
Disallow: /no/
Disallow: /ny/
Disallow: /or/
Disallow: /ps/
Disallow: /fa/
Disallow: /pl/
Disallow: /pt/
Disallow: /pa/
Disallow: /ro/
Disallow: /ru/
Disallow: /sm/
Disallow: /gd/
Disallow: /sr/
Disallow: /st/
Disallow: /sn/
Disallow: /sd/
Disallow: /si/
Disallow: /sk/
Disallow: /sl/
Disallow: /so/
Disallow: /es/
Disallow: /su/
Disallow: /sw/
Disallow: /sv/
Disallow: /tl/
Disallow: /tg/
Disallow: /ta/
Disallow: /te/
Disallow: /th/
Disallow: /tr/
Disallow: /tk/
Disallow: /uk/
Disallow: /ur/
Disallow: /ug/
Disallow: /uz/
Disallow: /vi/
Disallow: /cy/
Disallow: /xh/
Disallow: /yi/
Disallow: /yo/
Disallow: /zu/

-------------------------------------------------------------------
GROUP 2: COMMON CRAWL & DATASETS
-------------------------------------------------------------------
Large-scale crawlers used for AI training datasets.
High bandwidth consumption - monitor server load.
If performance issues arise, change Allow to Disallow.

User-agent: CCBot
User-agent: Diffbot
Allow: /
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /feed/
# Block Translated Locales (same bandwidth protection as Group 1B)
Disallow: /af/
Disallow: /sq/
Disallow: /am/
Disallow: /ar/
Disallow: /hy/
Disallow: /az/
Disallow: /eu/
Disallow: /be/
Disallow: /bn/
Disallow: /bs/
Disallow: /bg/
Disallow: /ca/
Disallow: /ceb/
Disallow: /zh-CN/
Disallow: /zh-TW/
Disallow: /co/
Disallow: /hr/
Disallow: /cs/
Disallow: /da/
Disallow: /nl/
Disallow: /eo/
Disallow: /et/
Disallow: /fi/
Disallow: /fr/
Disallow: /fy/
Disallow: /gl/
Disallow: /ka/
Disallow: /de/
Disallow: /el/
Disallow: /gu/
Disallow: /ht/
Disallow: /ha/
Disallow: /haw/
Disallow: /he/
Disallow: /hi/
Disallow: /hmn/
Disallow: /hu/
Disallow: /is/
Disallow: /ig/
Disallow: /id/
Disallow: /ga/
Disallow: /it/
Disallow: /ja/
Disallow: /jw/
Disallow: /kn/
Disallow: /kk/
Disallow: /km/
Disallow: /rw/
Disallow: /ko/
Disallow: /ku/
Disallow: /ky/
Disallow: /lo/
Disallow: /la/
Disallow: /lv/
Disallow: /lt/
Disallow: /lb/
Disallow: /mk/
Disallow: /mg/
Disallow: /ms/
Disallow: /ml/
Disallow: /mt/
Disallow: /mi/
Disallow: /mr/
Disallow: /mn/
Disallow: /my/
Disallow: /ne/
Disallow: /no/
Disallow: /ny/
Disallow: /or/
Disallow: /ps/
Disallow: /fa/
Disallow: /pl/
Disallow: /pt/
Disallow: /pa/
Disallow: /ro/
Disallow: /ru/
Disallow: /sm/
Disallow: /gd/
Disallow: /sr/
Disallow: /st/
Disallow: /sn/
Disallow: /sd/
Disallow: /si/
Disallow: /sk/
Disallow: /sl/
Disallow: /so/
Disallow: /es/
Disallow: /su/
Disallow: /sw/
Disallow: /sv/
Disallow: /tl/
Disallow: /tg/
Disallow: /ta/
Disallow: /te/
Disallow: /th/
Disallow: /tr/
Disallow: /tk/
Disallow: /uk/
Disallow: /ur/
Disallow: /ug/
Disallow: /uz/
Disallow: /vi/
Disallow: /cy/
Disallow: /xh/
Disallow: /yi/
Disallow: /yo/
Disallow: /zu/

-------------------------------------------------------------------
GROUP 3: SEO TOOLS (Rate Limited)
-------------------------------------------------------------------
Note: Crawl-delay respected by Bing/Yandex, ignored by Google.
wp-json blocked to limit competitor intelligence gathering.

User-agent: AhrefsBot
User-agent: SemrushBot
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /wp-json/
Disallow: /?s=

User-agent: MJ12bot
Crawl-delay: 20
Disallow: /wp-admin/
Disallow: /wp-json/
Disallow: /?s=

-------------------------------------------------------------------
GROUP 4: DEFAULT RULES (All Other Crawlers)
-------------------------------------------------------------------
Fallback rules for any crawler not explicitly listed above.
Note: Crawl-delay ignored by Google, respected by Bing/Yandex.

User-agent: *
Crawl-delay: 1
# Allow critical assets for proper page rendering
Allow: /*.css$
Allow: /*.js$
Allow: /wp-content/uploads/
# Block Admin & Core WordPress Files
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /readme.html
Disallow: /license.txt
# Block Duplicate Content & Efficiency Killers
Disallow: /?s=
Disallow: /*?s=
Disallow: /*?replytocom
Disallow: /trackback/
Disallow: /*?utm_
Disallow: /*?share=
Disallow: /feed/
Disallow: /comments/feed/

-------------------------------------------------------------------
SITEMAP
-------------------------------------------------------------------

Sitemap: https://ecommercefastlane.com/sitemap_index.xml
Sitemap: https://ecommercefastlane.com/llms.txt