# ============================================ # AI Language Models # ============================================ # LLM-optimized content: https://www.o-ring-stocks.eu/llms.txt User-agent: GPTBot Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Disallow: *?filter* Disallow: *?s=* Disallow: *?q=* Crawl-delay: 10 User-agent: ChatGPT-User Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 10 User-agent: ClaudeBot Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 10 User-agent: anthropic-ai Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 15 User-agent: PerplexityBot Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 10 User-agent: Amazonbot Allow: /llms.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 15 User-agent: CCBot Allow: /llms.txt Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 15 User-agent: meta-externalagent Allow: /llms.txt Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 10 User-agent: Applebot-Extended Allow: /llms.txt Crawl-delay: 10 User-agent: Bytespider Allow: /llms.txt Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Crawl-delay: 15 User-agent: cohere-ai Allow: /llms.txt Crawl-delay: 15 # ============================================ # Search Engines # ============================================ User-agent: Google* Allow: / User-agent: Google-InspectionTool Allow: / Crawl-delay: 5 User-agent: Googlebot Allow: / Crawl-Delay: 5 User-agent: Googlebot-Image Allow: / Crawl-Delay: 5 User-agent: Googlebot-News Crawl-Delay: 5 User-agent: Googlebot-Video Crawl-Delay: 5 User-agent: Bingbot Crawl-Delay: 10 # ============================================ # Ad Bots # ============================================ # ============================================ # Social Media # ============================================ User-agent: facebookexternalhit Crawl-delay: 10 User-agent: InstagramBot Crawl-delay: 10 # ============================================ # SEO Tools (rate limited) # ============================================ User-agent: SemrushBot Crawl-delay: 10 User-agent: SiteAuditBot Crawl-delay: 10 User-agent: AhrefsBot Crawl-Delay: 10 # ============================================ # Blocked Bots # ============================================ User-agent: PetalBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot/1.0 Disallow: / User-agent: Scrapy Disallow: / User-agent: HTTrack Disallow: / User-agent: python-requests Disallow: / User-agent: curl Disallow: / User-agent: wget Disallow: / User-agent: Go-http-client Disallow: / User-agent: okhttp Disallow: / User-agent: newspaper/0.2.8 Disallow: / User-agent: InfoTigerBot Disallow: / User-agent: IsraBot Disallow: / User-agent: Qwantify/2.4w Disallow: / User-agent: proximic Disallow: / User-agent: Orthogaffe Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: eCatch Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebReaper Disallow: / User-agent: Baiduspider Disallow: / User-agent: CazoodleBot Disallow: / User-agent: Fasterfox Disallow: / User-agent: Jyxobot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Indy Library Disallow: / User-agent: NetcraftSurveyAgent Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Python/3.9 aiohttp/3.8.1 Disallow: / # ============================================ # Slow Crawlers (rate limited) # ============================================ User-agent: Ingrid Crawl-delay: 20 User-agent: msnbot Crawl-Delay: 20 User-agent: Slurp Crawl-delay: 20 User-agent: Yahoo! Slurp Crawl-delay: 20 # ============================================ # Default rules for all other bots # ============================================ User-agent: * Disallow: */checkout/ Disallow: */rentalcheckout/ Disallow: */order/ Disallow: */loggedin/ Disallow: *?filter* Disallow: *?s=* Disallow: *?q=* Disallow: *?_gl=* Disallow: *?getFormHtmlOnly* Disallow: *?formId=* Disallow: *?pageId Disallow: *?cat_id= Disallow: *?question* Disallow: *spec= Disallow: *spec[ Disallow: *openSoftCompare Disallow: *material= Disallow: *brand= Disallow: *model= Disallow: *length= Disallow: *color= Disallow: *type= Disallow: *p= Disallow: *authorId= Disallow: /search/ Disallow: /upload/scripts/ Disallow: /specifications-pdf Disallow: /food-checkpostalcode Disallow: *ajax.php Disallow: *utm_medium=tradetracker Crawl-Delay: 10 Sitemap: https://www.o-ring-stocks.eu/sitemap-index.xml Sitemap: https://www.o-ring-stocks.eu/nl/sitemap-index.xml Sitemap: https://www.o-ring-stocks.eu/de/sitemap-index.xml Sitemap: https://www.o-ring-stocks.eu/fr/sitemap-index.xml Sitemap: https://www.o-ring-stocks.eu/it/sitemap-index.xml Sitemap: https://www.o-ring-stocks.eu/es/sitemap-index.xml