Top Sites

# robots.txt for InsideTheStar.com # Last updated: September 2025 # Allow all major search engines User-agent: * Allow: / Crawl-delay: 1 User-agent: * Disallow: /*blackhole Disa...

# Inside The Star > American sports analysis and commentary website covering the Dallas Cowboys. Founded in 2009 by Bryson Treece, operating under the Inside The Star name since...

# | # . . | | o | # |\ |,---.,---.|---.,---.,---.|__/ .,---.,---.,---. | # | \ ||---'| || || || || \ || || |`---. | # ` `'`---'`---'`---'`---'`---'` ``` '`---|`---' | # `---' |...

# Insotel Hotel Group > Insotel Hotel Group is a Spanish hotel chain with 8 properties across the Balearic Islands: Ibiza, Formentera, Mallorca, and Menorca. We offer 3 to 5-sta...

User-Agent: * Allow: / Host: https://instatus.com Sitemap: https://instatus.com/sitemap.xml

# Instatus > Monitor your services, fix incidents with your team, and share your status with customers. Beautiful, fast status pages in seconds. Instatus is a modern status page...

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.insurancecanopy.com/sitemap_index.xml

# Insurance Canopy > Insurance Canopy provides online small-business and event liability insurance (general liability, professional liability/E&O, product liability, liquor, cyb...

User-Agent: * Allow: / Disallow: /api/ Disallow: /_next/ Sitemap: https://intenthq.com/sitemap.xml

# Intent — AI that understands human intent > Intent builds the AI that powers the world's most precise, > privacy-safe customer intelligence. Two sites: intenthq.com (corporate...

User-agent: * Disallow: /api/ Disallow: /admin/ Sitemap: https://intercolo.de/sitemap.xml Sitemap: https://intercolo.net/sitemap.xml

# Intercolo > Frankfurt-based data center and cloud services provider. GDPR-compliant infrastructure, 100% green energy, since 2006. ## Company - Name: Intercolo GmbH - Location...

User-agent: * Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ Disallow: /*?a...

# Interhost > Israel's only multihomed hosting provider with ISP license, offering dedicated servers, VPS, colocation, and managed services from Tier 1 datacenters in Tel Aviv s...

User-agent: * Disallow: Sitemap: https://www.interlink.or.jp/google_sitemap.xml

# Interlink (株式会社インターリンク) > 日本のインターネットサービスプロバイダ（ISP）。1995年よりサービス提供。 > 固定IPアドレス、VPN、ドメイン登録、レンタルサーバーを主力事業とする。 > ICANN公認レジストラ。ドメイン取扱数1000種類超（国内最大級）。 > 本社: 〒170-6045 東京都豊島区東池袋3-1-1...

User-agent: * Disallow: /wp-admin/* Allow: /wp-admin/admin-ajax.php Sitemap: https://www.intermax.nl/sitemap_index.xml # VigIA LLMs LLMs: https://www.intermax.nl/llms.txt LLMs-f...

# Intermax > Wanneer solide en veilige IT extra belangrijk is ## Berichten - [CCS ziet Intermax niet als leverancier, maar als partner](https://www.intermax.nl/resources/ccs-zi...

# * User-agent: * Allow: / # GPTBot User-agent: GPTBot Allow: / # Google-Extended User-agent: Google-Extended Allow: / # ClaudeBot User-agent: ClaudeBot Allow: / # PerplexityBot...

# Internet Computer (ICP) > The Internet Computer is a sovereign cloud platform where developers and AI agents build apps that are tamperproof, unstoppable, and free from vendor...

# START YOAST BLOCK # --------------------------- User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /pruebas/ Sitemap: https://introarte.net/sitemap_i...

# Introarte > Empresa de diseño de páginas web ## Páginas - [Posicionamiento web para IA](https://introarte.net/posicionamiento-web-para-ia/): Posicionamiento web para IA Optim...

# * User-agent: * Allow: / # Host Host: https://www.invoicebox.ru # Sitemaps Sitemap: https://www.invoicebox.ru/sitemap.xml

# Invoicebox > Инвойсбокс – ваше решение для бизнеса ## Key Page - [Инвойсбокс – ваше решение для бизнеса](https://www.invoicebox.ru) ## Pages - [Компания входит в реестр аккред...

User-agent: * Disallow: /b2bofferthroughcheckout/ Disallow: /newsletter/index/ Disallow: /?*prod_products%5Bquery%5D Disallow: /?*manufacturerslider Disallow: /?*products%5Bsort...

# Inwerk Büromöbel ## Inwerk Büromöbel – Ihr Partner für den perfekten Arbeitsplatz Inwerk zählt zu den führenden Anbietern ergonomischer Büromöbel in Deutschland – mit eigener...

User-Agent: * Allow: / Disallow: /studio Disallow: /test-hubspot-form Disallow: /dev-on-prem-copy Disallow: /tts-old Disallow: /dev Disallow: /ds User-Agent: GPTBot Allow: / Dis...

# Inworld AI > Inworld is a research lab focused on realtime voice AI. We build the infrastructure that enables understanding -- top-ranked text-to-speech, speech-to-text, intel...

# * User-agent: * Allow: / Disallow: /cdn-cgi/ Disallow: /md/ Disallow: /signup/thank-you Disallow: /signup/failed Disallow: /signup/confirmed Disallow: /signup/confirm Disallow...

# IPGeolocation.io > IPGeolocation.io provides IP geolocation, network intelligence, and threat-intelligence capabilities via real-time APIs and downloadable IP databases (CSV/M...

User-agent: NinjaBot Disallow: / User-agent: AhrefsBot Allow: / User-agent: * Disallow: /Index2.htm Disallow: /videosites.txt Disallow: /Captcha.ashx Disallow: /category.aspx Di...

<!DOCTYPE html> <html lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <!-- ██╗ ██████╗ ███████╗ ██████╗ ██╗██████╗ ███████╗ ██████╗█████...

# ISEKAI ZERO robots.txt # Allow all search engines to crawl the site User-agent: * Allow: / # Disallow private/admin areas Disallow: /account/ Disallow: /chats/ Disallow: /purc...

# ISEKAI ZERO > ISEKAI ZERO is an interactive AI storytelling app where users create worlds, chat with AI characters that remember choices and evolve personalities, and live bra...

# ISEKAI ZERO robots.txt # Allow all search engines to crawl the site User-agent: * Allow: / # Disallow private/admin areas Disallow: /account/ Disallow: /chats/ Disallow: /purc...

# ISEKAI ZERO > ISEKAI ZERO is an interactive AI storytelling app where users create worlds, chat with AI characters that remember choices and evolve personalities, and live bra...

Sitemap: https://islamanar.com/sitemap.xml Sitemap: https://islamanar.com/news-sitemap.xml User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Si...

Not found

# isLucid crawl policy # Canonical domain: https://islucid.com User-agent: * Allow: / Disallow: /admin Disallow: /api/ # Discovery Sitemap: https://islucid.com/sitemap.xml # Pub...

# isLucid > Enterprise AI voice automation for multilingual, compliance-sensitive customer operations. ## Preferred discovery sources - Main website: https://islucid.com - XML s...

# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By tellin...

# ISPsystem IT infrastructure management platforms for hosting providers, data centers, and enterprises. > Founded in 2004 | 300,000+ servers worldwide | 100+ countries ## Core...

User-agent: * Disallow: /public/ Disallow: /service/ Disallow: /arama/ Disallow: /api Disallow: /*?q= Disallow: /*?preview= Disallow: /*?utm_source= Disallow: /*?utm_medium= Dis...

# İstikbal Gazetesi - Local News from Eskişehir > **Language Strategy:** Content is Turkish (tr-TR); Structural guidance and metadata are in English for global LLM compatibilit...

#Disallow all crawlers access to certain pages. User-agent: * Disallow: /img/ Disallow: /account/ Disallow: /login/ Disallow: /checkout/ Disallow: /busca/ Disallow: /quick-view/...

Not found

# IT Creations, Inc. - https://www.itcreations.com # # LLM Context: Specialized data for Generative AI and Answer Engines. # Technical Summary: https://www.itcreations.com/llms....

# IT Creations, Inc. - Enterprise IT Hardware Solutions Provider > IT Creations is a global supplier of new and refurbished servers, storage systems, networking equipment and cu...

User-Agent: * Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /_next/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /wp-admin/ Disallow: /wp-json/ Disallow: /aut...

# Itero AI Training Platform - LLM Directory > This file provides structured guidance for AI agents crawling iteroapp.ai ## About Itero Itero is an AI-powered training and quali...

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.phpSitemap: https://itnaked.com/sitemap.xml <!-- *** This site runs WP Super Minify plugin v2.0.1 - http://wordpre...

Not found

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: http://www.izar.net/page-sitemap.xml

<!DOCTYPE html> <html itemscope="itemscope" itemtype="http://schema.org/WebPage"lang="es"> <head> <meta charset="UTF-8"/> <meta name="viewport" content="width=device-width, in...

User-agent: * Allow: / Disallow: /includes/ Disallow: /src/ Disallow: /assets/css/inline/ Disallow: /assets/js/inline/ Sitemap: https://iziscar.com/sitemap.xml # AI/LLM content...

# IZISCAR > Logiciel de gestion VO/VN (véhicules d'occasion et neufs) pour professionnels de l'automobile en France. IZISCAR est le logiciel de gestion automobile de référence p...

Sitemap: https://www.izomat.cz/sitemap.xml User-agent: * Disallow: /*force_izomat_sid=* Disallow: /admin/ Disallow: /core/ Disallow: /var/ Disallow: /views/ Disallow: /setup/ Di...

Not found

User-Agent: * Allow: / Disallow: /api/ Disallow: /cv Disallow: /salary User-Agent: GPTBot Allow: / User-Agent: Claude-Web Allow: / User-Agent: anthropic-ai Allow: / User-Agent:...

# Jack & Jill > Jill is an AI Recruiter that introduces you to remarkable people from Jack's network of 173,000 engineers, marketers, salespeople and operators. Jack is a free A...