Top Sites

User-agent: * Disallow: /redirect Disallow: /api/ Disallow: /deals Disallow: /agent/ Disallow: /watching Disallow: /watched-searches Disallow: /alerts Disallow: /settings Disall...

# ShopSavvy > ShopSavvy is a price comparison platform that helps consumers find the best deals. We provide product lookups, real-time pricing from tens of thousands of retailer...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# rvtfloripa > O site sim.digital apresenta os serviços de internet banda larga 100% fibra óptica oferecidos pela empresa em Florianópolis. A sim.digital destaca a ultravelocida...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# simeona > O site da Simeona oferece uma gama completa de serviços de consultoria e sistemas de gestão para micro e pequenas empresas (MPEs). A empresa auxilia em áreas cruciai...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# landing > The provided content appears to be a technical reference for Google Publisher Tag and Ad Exchange tags. It lists different tag types, including standard Publisher Ta...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# siusag > Die SIUS AG präsentiert sich als weltweit führender Hersteller von elektronischen Trefferanzeigesystemen für Sport-, Jagd- und Militärschießen. Die Website bietet ein...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# skaarhoj > This website provides comprehensive information about SKAARHOJ's range of universal broadcast control panels and related software solutions. The site details their...

# Stockfilm robots.txt User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /boards Disallow: /challenge Disallow: /license Disallow: /rough-cut Disallow: /search...

# Stockfilm > Stockfilm is an archival footage licensing platform focused on restored 8mm and Super 8 home movie film. Buyers search 217,560 clips from the 1930s through the 198...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# sunstudio > Sun Studio, famously known as the "Birthplace of Rock'n'roll," offers visitors a historical journey through its iconic music legacy. The site provides daily guided...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# symphox > 神坊資訊股份有限公司（Symphox）提供企業數位轉型解決方案，核心服務涵蓋雲端與資訊安全、網路通訊基礎設施以及數據分析。該網站詳細介紹了公司在混合雲、網路建置、IDC 機房、API 安全、網站防護等多方面的專業顧問與技術服務。此外，網站也展示了其為企業打造的人才永續經營平台「有福企®」以及會員點數服務「小樹生活®」。神坊資訊致...

# Allow essential directories first User-agent: * Allow: /blog/ Allow: / # Disallow specific pages/directories User-agent: * Disallow: *?lightbox= Disallow: /_api/* Disallow: /_...

# synlabs > SynergyLabs Technology offers a broad spectrum of IT solutions, specializing in areas such as AI/ML, data science, and digital marketing. The site details their expe...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# moondancenetwork > Tanssi provides a decentralized infrastructure protocol for launching application-specific blockchains (Appchains), offering Ethereum-grade security and cus...

# AI Data Scraper # https://darkvisitors.com/agents/ai2bot User-agent: AI2Bot Disallow: / # AI Search Crawler # https://darkvisitors.com/agents/amazonbot User-agent: Amazonbot D...

{"@context":["https://www.w3.org/ns/activitystreams","https://w3id.org/security/v1","https://purl.archive.org/socialweb/webfinger",{"schema":"http://schema.org#","toot":"http://...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# my-site-1 > O site apresenta o tecpet, um sistema completo de gestão para pet shops, clínicas veterinárias e salões de banho e tosa. Ele oferece uma inteligência artificial in...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# 4pass-vpo-tech4log > O site da Tech4Log detalha suas políticas de cookies, privacidade, ética, termos e condições, e envio, explicando as práticas da empresa e os direitos dos...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# my-site > Testpapers is a South African educational platform offering a comprehensive collection of past exam papers, study guides, and summaries for students from Grade 1 to...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# texco2016 > Texla Control y Texco Solutions ofrecen una amplia gama de servicios tecnológicos para la modernización de redes y la transformación digital de empresas. El sitio...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# smartq > SmartQ is a B2B food experience company that transforms institutional food spaces like corporate cafeterias, university dining halls, and events through technology. T...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# my-site3 > TimeBanks.Org is a platform dedicated to the global TimeBanking movement, which builds stronger communities through reciprocal, time-based exchanges where one hour...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# my-site > This website provides a detailed course on eBay dropshipping, covering essential steps from account setup and product sourcing to order fulfillment and account manag...

# Trakkr Robots.txt # https://trakkr.ai # ============================================================================ # AI + SEARCH CRAWLERS - Allow public content, block priva...

# Trakkr > The AI visibility platform that helps brands understand, track, and improve how they appear in ChatGPT, Claude, Gemini, Perplexity, and other AI systems. ## Company O...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# tres-website > The TRES Technologies website offers a suite of integrated software solutions designed to streamline operations for travel agencies. It details products such as...

# ============================================================ # Trossen Robotics — robots.txt # Last updated: April 2026 # =====================================================...

# trossen-robotics > Trossen Robotics provides advanced AI hardware and robotic solutions, evolving from the ALOHA Project to make machine learning research more accessible thro...

# Gravity Website - robots.txt # https://www.trygravity.ai # # All crawlers welcome, including AI search bots # (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended,...

# Gravity > Gravity is the AI-native advertising platform. It delivers contextual, high-intent sponsored suggestions inside AI conversations, creating a new ad channel for LLM p...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# twilightcreationsinc > This website belongs to Twilight Creations Inc., a board game publisher established in 2002. The site showcases a diverse catalog of board games, includ...

User-agent: * Allow: / Sitemap: https://www.umu.se/sitemapindex2.xml

{"contentLink":{"id":517088,"workId":0,"guidValue":"7ddf2936-ff4b-4ff4-9b90-9220099c8a7d","url":"/status/404/"},"name":"404","language":{"link":"/status/404/","displayName":"Swe...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# home > The Universal Peace Federation (UPF) is an international and interreligious network dedicated to achieving world peace through various initiatives and associations. It...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# joykitchen > Bu site, 2017 yılında kurulan ve Joy Kitchen markasıyla ahşap züccaciye ürünleri üreten UP Mağazacılık'ı tanıtmaktadır. Düzce'deki modern üretim tesisleri, kalite...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# website-3 > USTA Northern California is dedicated to promoting and growing tennis across its region, aiming to make the sport accessible and enjoyable for everyone. The organi...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# vaultline ## AI Agent Access This site is powered by Wix and supports the Model Context Protocol (MCP) for agentic AI access. AI agents can connect directly to retrieve live,...

User-agent: * Allow: / Disallow: *?lightbox= # Optimization for Google Ads Bot User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google Disallow: /_partials* Disallow: /pro-ga...

# vitalitydentalarts > Vitality Dental Arts is a US-based dental laboratory that provides a wide array of dental restorations, including custom implant abutments, various types...