Top Sites>SSend deeply researched, high converting sales outreach | AiSDR

# START YOAST BLOCK
# ---------------------------
# robots.txt — AISDR
# Goal: allow indexing & AI-answer surfacing, block model-training harvesters.

Crawl-delay: 10

# ------------------------------
# Global rules (apply to ALL bots)
User-agent: *
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
Disallow: /admin/
Disallow: /internal/
Allow: /

# AI search/retrieval crawlers 
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Block AI model-training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /
User-agent: Bytespider
Disallow: /

# ------------------------------
# Traditional search engines (allowed for SEO + AI answer surfacing)
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

Sitemap: https://aisdr.com/sitemap_index.xml


# ------------------------------
# Sitemap
Sitemap: https://aisdr.com/sitemap_index.xml"
# ---------------------------
# END YOAST BLOCK

# LLM.txt — AISDR
# Purpose: Tell AI systems how they may use our content.
# This complements robots.txt (which controls crawling). LLM.txt focuses on usage rights.
# ---------------------
# Allowed Uses (Retrieval & Answers)
Policy: You MAY access and use this site’s publicly available content
for search, retrieval, indexing, and surfacing short excerpts in AI answers
(e.g., ChatGPT, Perplexity, Claude, Bing Copilot, Google AI Overviews, Apple Siri AI).
# ---------------------
# Disallowed Uses (Training & Datasets)
Policy: You MAY NOT use this content for model training, dataset creation, fine-tuning,
embeddings-for-training corpora, or any similar derivative dataset, whether
directly or via third parties or affiliates.
# ---------------------
# Respect robots.txt
Policy: Obey our robots.txt directives, including blocks for model-training endpoints
(e.g., GPTBot, Google-Extended, Applebot-Extended, ClaudeBot).
If your system has separate user agents for training vs. retrieval,
treat training agents as disallowed and retrieval agents as allowed.
# ---------------------
# Attribution
Policy: If you surface our content in answers, provide clear attribution to “AISDR”
with a visible link to
https://aisdr.com for the referenced page.
# ---------------------
# Rate & Freshness
Policy: Cache courteously and refresh periodically to avoid stale or misleading excerpts.
Excessive request rates may be blocked.
# ---------------------