Machine Readiness
Stored receipt and evidence
30
100
0
0
0
Samples
No stored offer samples.
Samples
No stored action samples.
Samples
No stored product samples.
Document
# ============================================================================= # AI Crawlers — Rate-Limited (Crawl-delay) # ============================================================================= # Дата: 2026-02-04, инцидент BUG-INFRA-003 # Причина: AI-боты генерировали 90%+ трафика, убивая серверы # Решение: Crawl-delay в robots.txt + rate limiting в nginx # # Crawl-delay: секунды между запросами (уважают Bing, Yandex, некоторые AI-боты) # Основное ограничение — nginx rate limiting (ai-bot-ratelimit.conf) # ============================================================================= # OpenAI User-agent: GPTBot Crawl-delay: 10 Allow: /catalog-avto/ Allow: /catalog-avto-china/ Allow: /catalog-avto-europe/ Allow: /catalog-avto-korea/ Allow: /blog Allow: /ratings Disallow: /api/ Disallow: /backend/ Disallow: /admin/ Disallow: /s/ User-agent: ChatGPT-User Crawl-delay: 10 Allow: / User-agent: OAI-SearchBot Crawl-delay: 5 Allow: / # Anthropic User-agent: ClaudeBot Crawl-delay: 10 Allow: /catalog-avto/ Allow: /catalog-avto-china/ Allow: /catalog-avto-europe/ Allow: /catalog-avto-korea/ Allow: /blog Allow: /ratings Disallow: /api/ Disallow: /backend/ Disallow: /admin/ Disallow: /s/ User-agent: Claude-SearchBot Crawl-delay: 5 Allow: / User-agent: Claude-User Crawl-delay: 5 Allow: / # Meta/Facebook User-agent: Meta-ExternalAgent Crawl-delay: 10 Allow: / User-agent: Meta-ExternalFetcher Crawl-delay: 10 Allow: / User-agent: FacebookBot Crawl-delay: 5 Allow: / # Amazon User-agent: Amazonbot Crawl-delay: 10 Allow: / # Perplexity User-agent: PerplexityBot Crawl-delay: 10 Allow: / User-agent: Perplexity-User Crawl-delay: 5 Allow: / # ByteDance / TikTok User-agent: Bytespider Crawl-delay: 15 Allow: / # Huawei User-agent: PetalBot Crawl-delay: 15 Allow: / # Google AI (не путать с Googlebot) User-agent: Google-CloudVertexBot Crawl-delay: 5 Allow: / # Apple User-agent: Applebot Allow: / # Common Crawl User-agent: CCBot Crawl-delay: 15 Allow: / # DuckDuckGo User-agent: DuckAssistBot Allow: / # Mistral User-agent: MistralAI-User Crawl-delay: 15 Allow: / # SEO-боты — сильно ограничиваем (не приносят прямого трафика) User-agent: SemrushBot Crawl-delay: 30 Disallow: /api/ Allow: / User-agent: AhrefsBot Crawl-delay: 30 Disallow: /api/ Allow: / User-agent: DotBot Crawl-delay: 30 Allow: / User-agent: DataForSeoBot Crawl-delay: 30 Allow: / User-agent: MJ12bot Crawl-delay: 30 Allow: / # Other AI bots User-agent: Anchor Browser User-agent: Novellum AI Crawl User-agent: ProRataInc User-agent: Timpibot Crawl-delay: 30 Allow: / # Internet Archive User-agent: archive.org_bot Allow: / Crawl-delay: 10 # ============================================================================= # Search Engine Crawlers # ============================================================================= User-agent: Googlebot User-agent: Bingbot Allow: / # Language parameter from Flutter app - block from indexing Disallow: /*?lang=* Disallow: /*&lang=* # Faceted Navigation - block filter combinations to save crawl budget # Google Dec 2024: https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation # Allow clean paths (make/model), block any URL with query parameters (filters) # Using *? pattern to match any path level + query string Allow: /catalog-avto/ Disallow: /catalog-avto*? Allow: /catalog-avto-china/ Allow: /catalog-avto-china-new/ Allow: /catalog-avto-china-used/ Disallow: /catalog-avto-china*? Disallow: /catalog-avto-china-new*? Disallow: /catalog-avto-china-used*? Allow: /catalog-avto-europe/ Disallow: /catalog-avto-europe*? Allow: /catalog-avto-korea/ Allow: /catalog-avto-korea-new/ Allow: /catalog-avto-korea-used/ Disallow: /catalog-avto-korea*? Disallow: /catalog-avto-korea-new*? Disallow: /catalog-avto-korea-used*? Allow: /catalog-avto-uae/ Disallow: /catalog-avto-uae*? # Powersports catalogs - block query params (pagination, filters) Allow: /catalog-motorcycles/ Disallow: /catalog-motorcycles*? Allow: /catalog-atv/ Disallow: /catalog-atv*? Allow: /catalog-jet-ski/ Disallow: /catalog-jet-ski*? Allow: /catalog-snowmobiles/ Disallow: /catalog-snowmobiles*? Allow: /catalog-baggi/ Disallow: /catalog-baggi*? # Block invalid URLs where lot_number is used as make (e.g. /catalog-avto-china/472166/model-name) # Make names cannot start with numbers - these are invalid URLs that should not be indexed # Using * wildcard for explicit matching of all paths starting with digit Disallow: /catalog-avto/0* Disallow: /catalog-avto/1* Disallow: /catalog-avto/2* Disallow: /catalog-avto/3* Disallow: /catalog-avto/4* Disallow: /catalog-avto/5* Disallow: /catalog-avto/6* Disallow: /catalog-avto/7* Disallow: /catalog-avto/8* Disallow: /catalog-avto/9* Disallow: /catalog-avto-china/0* Disallow: /catalog-avto-china/1* Disallow: /catalog-avto-china/2* Disallow: /catalog-avto-china/3* Disallow: /catalog-avto-china/4* Disallow: /catalog-avto-china/5* Disallow: /catalog-avto-china/6* Disallow: /catalog-avto-china/7* Disallow: /catalog-avto-china/8* Disallow: /catalog-avto-china/9* Disallow: /catalog-avto-china-new/0* Disallow: /catalog-avto-china-new/1* Disallow: /catalog-avto-china-new/2* Disallow: /catalog-avto-china-new/3* Disallow: /catalog-avto-china-new/4* Disallow: /catalog-avto-china-new/5* Disallow: /catalog-avto-china-new/6* Disallow: /catalog-avto-china-new/7* Disallow: /catalog-avto-china-new/8* Disallow: /catalog-avto-china-new/9* Disallow: /catalog-avto-china-used/0* Disallow: /catalog-avto-china-used/1* Disallow: /catalog-avto-china-used/2* Disallow: /catalog-avto-china-used/3* Disallow: /catalog-avto-china-used/4* Disallow: /catalog-avto-china-used/5* Disallow: /catalog-avto-china-used/6* Disallow: /catalog-avto-china-used/7* Disallow: /catalog-avto-china-used/8* Disallow: /catalog-avto-china-used/9* Disallow: /catalog-avto-europe/0* Disallow: /catalog-avto-europe/1* Disallow: /catalog-avto-europe/2* Disallow: /catalog-avto-europe/3* Disallow: /catalog-avto-europe/4* Disallow: /catalog-avto-europe/5* Disallow: /catalog-avto-europe/6* Disallow: /catalog-avto-europe/7* Disallow: /catalog-avto-europe/8* Disallow: /catalog-avto-europe/9* Disallow: /catalog-avto-korea/0* Disallow: /catalog-avto-korea/1* Disallow: /catalog-avto-korea/2* Disallow: /catalog-avto-korea/3* Disallow: /catalog-avto-korea/4* Disallow: /catalog-avto-korea/5* Disallow: /catalog-avto-korea/6* Disallow: /catalog-avto-korea/7* Disallow: /catalog-avto-korea/8* Disallow: /catalog-avto-korea/9* Disallow: /catalog-avto-korea-new/0* Disallow: /catalog-avto-korea-new/1* Disallow: /catalog-avto-korea-new/2* Disallow: /catalog-avto-korea-new/3* Disallow: /catalog-avto-korea-new/4* Disallow: /catalog-avto-korea-new/5* Disallow: /catalog-avto-korea-new/6* Disallow: /catalog-avto-korea-new/7* Disallow: /catalog-avto-korea-new/8* Disallow: /catalog-avto-korea-new/9* Disallow: /catalog-avto-korea-used/0* Disallow: /catalog-avto-korea-used/1* Disallow: /catalog-avto-korea-used/2* Disallow: /catalog-avto-korea-used/3* Disallow: /catalog-avto-korea-used/4* Disallow: /catalog-avto-korea-used/5* Disallow: /catalog-avto-korea-used/6* Disallow: /catalog-avto-korea-used/7* Disallow: /catalog-avto-korea-used/8* Disallow: /catalog-avto-korea-used/9* Disallow: /catalog-avto-uae/0* Disallow: /catalog-avto-uae/1* Disallow: /catalog-avto-uae/2* Disallow: /catalog-avto-uae/3* Disallow: /catalog-avto-uae/4* Disallow: /catalog-avto-uae/5* Disallow: /catalog-avto-uae/6* Disallow: /catalog-avto-uae/7* Disallow: /catalog-avto-uae/8* Disallow: /catalog-avto-uae/9* # Short URLs - hidden from indexing (for messenger sharing only) Disallow: /s/ # System directories Disallow: /backend/ Disallow: /admin/ Disallow: /api/ Disallow: /vendor/ Disallow: /storage/ Disallow: /node_modules/ Disallow: /resources/ Disallow: /tests/ # ============================================================================= # Yandex - Special Rules # ============================================================================= User-agent: Yandex Allow: / Disallow: /*amp/ Disallow: /*?lang=* Disallow: /*&lang=* Disallow: /s/ Disallow: /backend/ Disallow: /admin/ Host: https://westmotors.by # Clean-param: объединение дублей страниц с GET-параметрами сортировки # Эти параметры не меняют контент, только порядок отображения # https://yandex.ru/support/webmaster/robot-workings/clean-param.html Clean-param: sort_by&sort_dir&tab /catalog-avto Clean-param: sort_by&sort_dir&tab /catalog-avto-china Clean-param: sort_by&sort_dir&tab /catalog-avto-china-new Clean-param: sort_by&sort_dir&tab /catalog-avto-china-used Clean-param: sort_by&sort_dir&tab /catalog-avto-korea Clean-param: sort_by&sort_dir&tab /catalog-avto-korea-new Clean-param: sort_by&sort_dir&tab /catalog-avto-korea-used Clean-param: sort_by&sort_dir&tab /catalog-avto-europe Clean-param: sort_by&sort_dir&tab /catalog-avto-uae Clean-param: sort_by&sort_dir&tab /catalog-motorcycles Clean-param: sort_by&sort_dir&tab /catalog-atv Clean-param: sort_by&sort_dir&tab /catalog-jet-ski Clean-param: sort_by&sort_dir&tab /catalog-snowmobiles Clean-param: sort_by&sort_dir&tab /catalog-baggi # ============================================================================= # Baidu # ============================================================================= User-agent: Baiduspider Allow: / Disallow: /backend/ Disallow: /admin/ # ============================================================================= # Default Rules (All Other Bots) # ============================================================================= User-agent: * Allow: / # Language parameter from Flutter app - block from indexing Disallow: /*?lang=* Disallow: /*&lang=* # Faceted Navigation - block filter combinations (any path + query params) Allow: /catalog-avto/ Disallow: /catalog-avto*? Allow: /catalog-avto-china/ Allow: /catalog-avto-china-new/ Allow: /catalog-avto-china-used/ Disallow: /catalog-avto-china*? Disallow: /catalog-avto-china-new*? Disallow: /catalog-avto-china-used*? Allow: /catalog-avto-europe/ Disallow: /catalog-avto-europe*? Allow: /catalog-avto-korea/ Allow: /catalog-avto-korea-new/ Allow: /catalog-avto-korea-used/ Disallow: /catalog-avto-korea*? Disallow: /catalog-avto-korea-new*? Disallow: /catalog-avto-korea-used*? Allow: /catalog-avto-uae/ Disallow: /catalog-avto-uae*? # Powersports catalogs - block query params (pagination, filters) Allow: /catalog-motorcycles/ Disallow: /catalog-motorcycles*? Allow: /catalog-atv/ Disallow: /catalog-atv*? Allow: /catalog-jet-ski/ Disallow: /catalog-jet-ski*? Allow: /catalog-snowmobiles/ Disallow: /catalog-snowmobiles*? Allow: /catalog-baggi/ Disallow: /catalog-baggi*? # Block invalid URLs where lot_number is used as make (e.g. /catalog-avto-china/472166/model-name) # Make names cannot start with numbers - these are invalid URLs that should not be indexed # Using * wildcard for explicit matching of all paths starting with digit Disallow: /catalog-avto/0* Disallow: /catalog-avto/1* Disallow: /catalog-avto/2* Disallow: /catalog-avto/3* Disallow: /catalog-avto/4* Disallow: /catalog-avto/5* Disallow: /catalog-avto/6* Disallow: /catalog-avto/7* Disallow: /catalog-avto/8* Disallow: /catalog-avto/9* Disallow: /catalog-avto-china/0* Disallow: /catalog-avto-china/1* Disallow: /catalog-avto-china/2* Disallow: /catalog-avto-china/3* Disallow: /catalog-avto-china/4* Disallow: /catalog-avto-china/5* Disallow: /catalog-avto-china/6* Disallow: /catalog-avto-china/7* Disallow: /catalog-avto-china/8* Disallow: /catalog-avto-china/9* Disallow: /catalog-avto-china-new/0* Disallow: /catalog-avto-china-new/1* Disallow: /catalog-avto-china-new/2* Disallow: /catalog-avto-china-new/3* Disallow: /catalog-avto-china-new/4* Disallow: /catalog-avto-china-new/5* Disallow: /catalog-avto-china-new/6* Disallow: /catalog-avto-china-new/7* Disallow: /catalog-avto-china-new/8* Disallow: /catalog-avto-china-new/9* Disallow: /catalog-avto-china-used/0* Disallow: /catalog-avto-china-used/1* Disallow: /catalog-avto-china-used/2* Disallow: /catalog-avto-china-used/3* Disallow: /catalog-avto-china-used/4* Disallow: /catalog-avto-china-used/5* Disallow: /catalog-avto-china-used/6* Disallow: /catalog-avto-china-used/7* Disallow: /catalog-avto-china-used/8* Disallow: /catalog-avto-china-used/9* Disallow: /catalog-avto-europe/0* Disallow: /catalog-avto-europe/1* Disallow: /catalog-avto-europe/2* Disallow: /catalog-avto-europe/3* Disallow: /catalog-avto-europe/4* Disallow: /catalog-avto-europe/5* Disallow: /catalog-avto-europe/6* Disallow: /catalog-avto-europe/7* Disallow: /catalog-avto-europe/8* Disallow: /catalog-avto-europe/9* Disallow: /catalog-avto-korea/0* Disallow: /catalog-avto-korea/1* Disallow: /catalog-avto-korea/2* Disallow: /catalog-avto-korea/3* Disallow: /catalog-avto-korea/4* Disallow: /catalog-avto-korea/5* Disallow: /catalog-avto-korea/6* Disallow: /catalog-avto-korea/7* Disallow: /catalog-avto-korea/8* Disallow: /catalog-avto-korea/9* Disallow: /catalog-avto-korea-new/0* Disallow: /catalog-avto-korea-new/1* Disallow: /catalog-avto-korea-new/2* Disallow: /catalog-avto-korea-new/3* Disallow: /catalog-avto-korea-new/4* Disallow: /catalog-avto-korea-new/5* Disallow: /catalog-avto-korea-new/6* Disallow: /catalog-avto-korea-new/7* Disallow: /catalog-avto-korea-new/8* Disallow: /catalog-avto-korea-new/9* Disallow: /catalog-avto-korea-used/0* Disallow: /catalog-avto-korea-used/1* Disallow: /catalog-avto-korea-used/2* Disallow: /catalog-avto-korea-used/3* Disallow: /catalog-avto-korea-used/4* Disallow: /catalog-avto-korea-used/5* Disallow: /catalog-avto-korea-used/6* Disallow: /catalog-avto-korea-used/7* Disallow: /catalog-avto-korea-used/8* Disallow: /catalog-avto-korea-used/9* Disallow: /catalog-avto-uae/0* Disallow: /catalog-avto-uae/1* Disallow: /catalog-avto-uae/2* Disallow: /catalog-avto-uae/3* Disallow: /catalog-avto-uae/4* Disallow: /catalog-avto-uae/5* Disallow: /catalog-avto-uae/6* Disallow: /catalog-avto-uae/7* Disallow: /catalog-avto-uae/8* Disallow: /catalog-avto-uae/9* # Short URLs - hidden from indexing (for messenger sharing only) Disallow: /s/ # System directories Disallow: /backend/ Disallow: /admin/ Disallow: /api/ Disallow: /*?2amo* Disallow: /cdn-cgi/l/email-protection Disallow: /vendor/ Disallow: /storage/ Disallow: /node_modules/ Disallow: /resources/ Disallow: /tests/ # Auth & service pages - no indexing needed Disallow: /login Disallow: /register Disallow: /password/reset Disallow: /thank-you # Container tracking - allow main page, disallow subpages Allow: /container-tracking Disallow: /container-tracking-map/* Disallow: /container-tracking/* # ============================================================================= # Sitemap # ============================================================================= Sitemap: https://westmotors.by/sitemap.xml # ============================================================================= # LLMs.txt (AI agent information files) # ============================================================================= # llms.txt — краткий обзор сайта для AI-агентов # llms-full.txt — полная информация о компании, услугах, каталогах, FAQ # Стандарт: https://llmstxt.org/ LLMs-txt: https://westmotors.by/llms.txt LLMs-full-txt: https://westmotors.by/llms-full.txt
Document
# WESTMOTORS > WESTMOTORS — международная компания по импорту автомобилей из США, Китая, Кореи, Европы и ОАЭ. На рынке с 2008 года. Офисы в Беларуси, России, Казахстане, Польше и развитая сеть по ЕС. Официальный партнёр Insurance Auto Auction (IAA). - [О компании](https://westmotors.by/about) - [Каталог авто из США](https://westmotors.by/catalog-avto): 100,000+ лотов Copart/IAAI - [Каталог авто из Китая](https://westmotors.by/catalog-avto-china): BYD, Li Auto, NIO, Zeekr, Geely - [Каталог новых авто из Китая](https://westmotors.by/catalog-avto-china-new) - [Каталог б/у авто из Китая](https://westmotors.by/catalog-avto-china-used) - [Каталог авто из Кореи](https://westmotors.by/catalog-avto-korea): Hyundai, Kia, Genesis - [Каталог авто из Европы](https://westmotors.by/catalog-avto-europe): Audi, BMW, Mercedes, VW - [Каталог авто из ОАЭ](https://westmotors.by/catalog-avto-uae): люксовые и спорткары - [Каталог мотоциклов](https://westmotors.by/catalog-motorcycles) - [Каталог квадроциклов](https://westmotors.by/catalog-atv) - [Каталог гидроциклов](https://westmotors.by/catalog-jet-ski) - [Каталог снегоходов](https://westmotors.by/catalog-snowmobiles) - [Каталог багги/UTV](https://westmotors.by/catalog-baggi) - [Калькулятор растаможки](https://westmotors.by/calculator): расчёт полной стоимости под ключ - [Отслеживание контейнеров](https://westmotors.by/container-tracking): статус доставки в реальном времени - [FAQ](https://westmotors.by/faq): частые вопросы об импорте - [FAQ: Авто из США](https://westmotors.by/faq/usa) - [FAQ: Авто из Китая](https://westmotors.by/faq/china) - [FAQ: Авто из Кореи](https://westmotors.by/faq/korea) - [FAQ: Авто из Европы](https://westmotors.by/faq/europe) - [FAQ: Авто из ОАЭ](https://westmotors.by/faq/uae) - [Блог](https://westmotors.by/blog): статьи, обзоры, руководства по импорту - [Контакты](https://westmotors.by/contacts) - [Отзывы](https://westmotors.by/testimonials) - [Официальная сделка](https://westmotors.by/avto-import) - [Аукционы США](https://westmotors.by/auctions) - [Аукционы Европы](https://westmotors.by/auctions-europe) - [Авто в наличии](https://westmotors.by/avto-v-nalichii) - [Полная версия для AI](https://westmotors.by/llms-full.txt) ## Контакты (Беларусь, Минск) - Телефон: +375 (29) 170-70-01 - Email: office@westmotors.by - Адрес: Минск, ул. Тимирязева, 114, ТЦ «Автоград», оф. 3064 - Режим работы: Пн-Пт: 9:00-18:00 | Сб-Вс: 10:00-16:00 - Telegram: tg://resolve?domain=OFFICE_WESTMOTORS - WhatsApp: https://wa.me/375291707001
Document
Not stored for this site.