Top Sites>BCC Research: Global Markets Reports and Industry Analysis

# BCC Research robots.txt — AI-friendly with guardrails

# --- Global defaults (keep existing hygiene) ---
User-agent: *
Crawl-Delay: 2
Disallow: /index/advancedresult/
Disallow: /index/advancedsearch/
Disallow: /*?*
Disallow: /*?
Disallow: /aboutus/hubspotlead
Disallow: /statsana/piwik/
Allow: /public/
Allow: /public/images2017/

# --- AI crawlers: explicit allow (faster crawl) ---
User-agent: GPTBot
Allow: /
Crawl-Delay: 2

User-agent: OAI-SearchBot
Allow: /
Crawl-Delay: 2

User-agent: ClaudeBot
Allow: /
Crawl-Delay: 2

User-agent: Claude-SearchBot
Allow: /
Crawl-Delay: 2

User-agent: PerplexityBot
Allow: /
Crawl-Delay: 2

User-agent: CCBot
Allow: /
Crawl-Delay: 2

# Broader “infra” user-agents used for AI indexing/pipeline work
User-agent: GoogleOther
Allow: /
Crawl-Delay: 2

User-agent: Amazonbot
Allow: /
Crawl-Delay: 2

# Traditional search bots (often feed LLM features)
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

# Training/usage policy toggles (not crawlers, but respected by some vendors)
User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# --- Machine Content Policy for AI & LLMs (already live) ---
MCP: https://www.bccresearch.com/mcp.yaml

# --- Sitemaps (already live) ---
Sitemap: https://www.bccresearch.com/public/sitemap/indexsitemap/sitemapindex.xml

# BCC Research LLM Policy (experimental)
# Location: https://www.bccresearch.com/llms.txt
# Purpose: Declare how Large Language Models and AI agents may access and use BCC content.
# NOTE: This file complements robots.txt and MCP; compliant AI systems should honor all three.

Policy-Version: 1.0
Updated: 2025-08-13
Owner: BCC Research
Website: https://www.bccresearch.com/
Contact: info@bccresearch.com
MCP: https://www.bccresearch.com/mcp.yaml
Robots: https://www.bccresearch.com/robots.txt

Scope:
  - Applies to publicly available content on bccresearch.com (including /public/ paths).
  - Excludes gated/purchased content and authenticated areas.

# -------- Access Rules (AI-friendly, align with robots.txt) --------
# Public access allowed, but respect the disallows below.

Allow: /

# Keep these areas off-limits to LLM crawling/ingestion:
Disallow: /aboutus/hubspotlead
Disallow: /statsana/piwik/

# Known AI user-agents (non-exhaustive). Others should follow the same rules.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: CCBot
User-agent: GoogleOther
User-agent: Amazonbot

# -------- Usage & Licensing Terms for AI Systems --------
Attribution: Required. Cite “Source: BCC Research — https://www.bccresearch.com/” and link to the specific page.
Summarization: Allowed for public pages with attribution.
Display: Short excerpts/snippets may be shown with attribution; do NOT reproduce full reports or paywalled content.
Storage/Caching: Do NOT store or redistribute full report text. Ephemeral caching of small excerpts/metadata ≤ 24 hours is acceptable for ranking and deduplication.
Model Training/Fine-tuning: PROHIBITED without a written license from BCC Research.
Derivatives: Non-commercial analytical summaries permitted with attribution; commercial derivatives require a license.
Rate: Be polite — ≤ 1 request/second, ≤ 2 concurrent requests per client. Excess load may be throttled or blocked.
Headers: Send a descriptive User-Agent and a Referer header (see MCP “compliance”).
Respect Robots: If this file and robots.txt differ, the more restrictive rule applies.

# -------- Authentication & Paywalls --------
Private, licensed, or logged-in content is out of scope for access, training, storage, or display. Do not attempt to bypass authentication or paywalls.

# -------- Preferred Machine Interface --------
Use of the official MCP endpoint is preferred for programmatic access to structured data:
MCP-Endpoint: https://public.mcp.bccresearch.com/mcp
MCP-Method: POST
MCP-Docs: https://www.bccresearch.com/mcp.yaml

# -------- Opt-out / Takedown --------
If you operate an AI system and believe you ingested BCC content in violation of this policy, or you wish to request a license:
Email: info@bccresearch.com (subject: “AI/Data Licensing”)

# -------- Changes --------
This policy may evolve. Check this file and the MCP policy for the latest terms.