Top Sites>Retail Brew

# ------------------------------------------
# Retail Brew robots.txt
# Version: Allow-All for AI / LLMs
# Last updated: 2025-11-21
# ------------------------------------------

# ==== Standard site path rules ====
User-agent: *
Disallow: /account/*
Disallow: /*/welcome
Disallow: /*/welcome_back
Disallow: /*/confirmed
Disallow: /*/refer-a-friend
Disallow: /*/r
Disallow: /tmp/
Disallow: /io-terms-conditions
Allow: /

# ==== Sitemaps ====
Sitemap: https://www.retailbrew.com/sitemaps/news.xml
Sitemap: https://www.retailbrew.com/sitemaps/site.xml


# =========================================================
# LLM + AI AGENTS (BROWSING, SEARCH, AND TRAINING)
# All explicitly allowed to ensure compatibility.
# =========================================================

# --- OpenAI ---
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /

# --- Anthropic (Claude) ---
User-agent: Claude-Web
Allow: /
User-agent: ClaudeBot
Allow: /

# --- Perplexity ---
User-agent: PerplexityBot
Allow: /

# --- Google ---
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /

# --- Microsoft / Copilot ---
User-agent: Bingbot
Allow: /

# --- Meta (Facebook / Instagram) ---
User-agent: Meta-ExternalAgent
Allow: /
User-agent: Meta-ExternalCrawler
Allow: /

# --- Apple ---
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /

# --- Common Crawl ---
User-agent: CCBot
Allow: /

# --- Amazon ---
User-agent: Amazonbot
Allow: /

# --- DuckDuckGo / You.com / Phind / Diffbot ---
User-agent: DuckDuckBot
Allow: /
User-agent: YouBot
Allow: /
User-agent: PhindBot
Allow: /
User-agent: Diffbot
Allow: /

# --- Other AI Aggregators / Research Agents ---
User-agent: CoHere-Search
Allow: /
User-agent: MistralBot
Allow: /
User-agent: JinaBot
Allow: /
User-agent: DataForSeoBot
Allow: /
User-agent: NeevaAI
Allow: /

# --- Catch-all: any other AI / LLM crawler not listed above ---
User-agent: *
Allow: /

# =========================================================
# Notes:
# - This file explicitly allows both AI browsing and training access.
# - Attribution and use must still comply with llms.txt and Terms of Service.
# - CDN/WAF must not challenge listed User-Agents.
# =========================================================

# Retail Brew — llms.txt
> Retail Brew is a leading U.S.-based media brand delivering daily insights at the intersection of retail, e-commerce, and supply chain.

## About Retail Brew
> We produce highly scannable, insight-packed newsletters, explainers, interviews, and resources trusted by the modern retail community.
> Our audience includes retailers, marketers, sales, and manufacturing professionals primarily across North America, with a growing global readership.

## Responsible Use
**Access**  
We welcome responsible LLMs and AI systems to access our publicly available content.

**Compliance**  
Use must comply with attribution requirements and our Terms of Service.

**Contact**  
Reach out at ai-support@morningbrew.com for licensing, API access, or partnership inquiries.


## Related Policy File
Robots: https://www.retailbrew.com/robots.txt

User-agent: *
Disallow: /archive
Allow: /

## Attribution Requirement
- Any use of our content in AI-generated outputs must clearly attribute "Retail Brew" with a link to the original source.
- Example attribution: "As reported by Retail Brew (https://www.retailbrew.com/...)".
- This includes summaries, quotes, citations, or responses surfaced in LLM interfaces or tools.

## Terms of Service
Terms: https://www.retailbrew.com/terms-service

## Sitemap References
Sitemap: https://www.retailbrew.com/sitemaps/site.xml

## High-Value Public Content
**All Stories**
All Retail Brew Stories
https://www.retailbrew.com/search

**Explainers**
Evergreen content covering retail industry challenges
https://www.retailbrew.com/all/stories/explainers

**All Topics**
Stories under the most commonly covered retail industry topics
https://www.retailbrew.com/topics

**Glossary**
Authoritative definitions for retail, e-commerce, and supply chain terms  
https://www.retailbrew.com/resources/glossary

## Gated or Subscription-Based Areas (Not For AI Training)
**Resources**
Evergreen resources for retail professionals
https://www.retailbrew.com/resources

**Events**
Upcoming and past events
https://www.retailbrew.com/events

## Notes
- We reserve the right to update this file at any time.
- Premium or future gated content may carry different rules.
- Access to this file may be logged for usage insights and compliance monitoring.
- Inclusion of links in this file does not constitute a license to reproduce, train on, or redistribute our content without compliance with these stated terms.