Top Sites>BeFreed: Personalized Audio Learning for Anything

# robots.txt for https://www.befreed.ai
# Updated: 2025-07-11

# =============================================================================
# Malicious/Unwanted Bot Blocking
# =============================================================================

# Content scrapers and downloaders
User-agent: HTTrack
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: Fetch
Disallow: /

# =============================================================================
# High-frequency/Misbehaving Crawlers (Complete Block)
# =============================================================================

User-agent: wget
User-agent: fast
User-agent: grub-client
User-agent: NPBot
User-agent: larbin
User-agent: libwww
User-agent: ZyBORG
User-agent: Zealbot
User-agent: UbiCrawler
User-agent: MSIECrawler
User-agent: Xenu
Disallow: /

# =============================================================================
# SEO Analysis Tools (Special Restrictions)
# =============================================================================

# SEO analysis tools - additional restrictions beyond general rules 
User-agent: SemrushBot
User-agent: AhrefsBot
User-agent: MJ12bot

# (lift for now)
# Disallow: /categories/ 
# Disallow: /collection/
# Disallow: /books/

# Higher crawl delay for analysis tools
Crawl-delay: 10

# =============================================================================
# All Other Bots (Including AI Search Engines & Traditional Search)
# =============================================================================

User-agent: *

# Allow important public content (explicit allow for SEO clarity)
Allow: /
Allow: /blog/
Allow: /collection/
Allow: /book/
Allow: /about
Allow: /faq
Allow: /privacy
Allow: /term
Allow: /categories/
Allow: /partnerships
Allow: /career
Allow: /download
Allow: /list/
Allow: /topics/
Allow: /directory/

# Authentication routes and proxies
Disallow: /auth/
Disallow: /finish_signin

# Premium content requiring subscription
Disallow: /book/*/deep
Disallow: /book/*/flash
Disallow: /book/*/heart
Disallow: /book/*/funs

# User-specific authenticated content
Disallow: /app/browse
Disallow: /app/journey
Disallow: /app/section
Disallow: /library/
Disallow: /memory
Disallow: /search
Disallow: /activity
Disallow: /playgorund_experimental

# API endpoints (protect backend)
Disallow: /api/
Disallow: /book/opengraph-image*

# Next.js internal routes (image optimization API)
Disallow: /_next/image
Disallow: /_next/static

# URL parameters that don't add SEO value
Disallow: /*?*utm_source=
Disallow: /*?*utm_medium=
Disallow: /*?*utm_campaign=
Disallow: /*?*utm_content=
Disallow: /*?*utm_term=
Disallow: /*?*showSignup=
Disallow: /*?*showLogin=
Disallow: /*?*modal=
Disallow: /*?*popup=
Disallow: /*?*auth=
Disallow: /*?*showPaywall=
Disallow: /*?*via=

# Dynamic route parameters that might cause duplicate content
Disallow: /*?*ref=
Disallow: /*?*share=
Disallow: /*?*sid=
Disallow: /*?*sessionid=
Disallow: /*?*token=

# Prevent indexing of common file types that shouldn't be searched
Disallow: /*.json$
Disallow: /*.log$

# Sitemap
Sitemap: https://www.befreed.ai/sitemap.xml

# This site provides an LLM access policy at: https://www.befreed.ai/llms.txt

# Crawl delay for all bots (prevent server overload)
Crawl-delay: 2

# BeFreed

> Premium educational platform providing expert-curated book summaries, key insights, and personalized learning experiences for professionals and lifelong learners seeking accelerated knowledge acquisition.

## Core Content

[Book Summaries](https://www.befreed.ai/book/): Expert-curated summaries of bestselling business, productivity, and personal development books. Each summary is 10-15 minutes read, professionally edited, and includes 5-7 actionable insights with implementation guidance. Updated weekly with new releases.

[Blog Articles](https://www.befreed.ai/blog/): In-depth articles on productivity, learning techniques, and professional development written by industry experts. Content focuses on practical strategies, scientific research, and real-world case studies for busy professionals.

[Book Collections](https://www.befreed.ai/collection/): Curated reading lists organized by topics such as leadership, entrepreneurship, productivity, and personal growth. Each collection includes 8-12 books with progression difficulty and estimated completion timeline.

[Categories](https://www.befreed.ai/categories/): Browse books by subject areas including business strategy, psychology, self-improvement, and professional skills. Content organized by expertise level (beginner to advanced) and industry application.

[Topics](https://www.befreed.ai/topics/): Thematic book recommendations featuring "Best of" lists for specific subjects such as management, trading, stoicism, and American history. Each topic page includes expert-selected titles with detailed explanations and background context from subject matter specialists.

[Reading Lists](https://www.befreed.ai/list/): Comprehensive book collections with detailed summaries, author overviews, key takeaways, and FAQs for each title. Interactive format allows deep exploration of book content including overview, author background, actionable insights, and frequently asked questions.

[Directory](https://www.befreed.ai/directory/): Alphabetically organized index of every BeFreed book summary. Browse titles via A–Z, 0–9, and a special “-” category using instant letter navigation. Utilizes server-streamed results for fast first paint and optimal SEO—perfect for users who prefer discovering content by book name rather than topic or collection.

[Mobile App Download](https://www.befreed.ai/download): Cross-platform mobile application available for iOS and Android devices. Features offline reading, personalized learning recommendations, and optimized mobile reading experience. Rated as #1 Smart Learning App 2025 with PWA technology for Android and native iOS app store distribution.

## Content Quality Standards

- **Expert Curation**: All content reviewed by subject matter experts and professional editors
- **Source Material**: Based on bestselling books from renowned authors and publishers
- **Actionable Focus**: Every summary includes practical takeaways and implementation steps
- **Time-Optimized**: Designed for busy professionals with 10-15 minute consumption windows
- **Regular Updates**: New content added weekly, existing content reviewed quarterly

## Target Audience

**Primary Users**: Working professionals, entrepreneurs, executives, and lifelong learners seeking efficient knowledge acquisition and skill development.

**Use Cases**: Career advancement, leadership development, productivity improvement, personal growth, and staying current with business trends.

## Company Information

[About BeFreed](https://www.befreed.ai/about): Mission to democratize access to high-quality educational content and accelerate professional development through curated learning experiences. Founded by learning experts with backgrounds in education technology and content curation.

[Partnership Opportunities](https://www.befreed.ai/partnerships): Collaboration opportunities for publishers, authors, educators, and corporate learning programs. Custom content creation and white-label solutions available.

[Careers at BeFreed](https://www.befreed.ai/career): Current job openings including engineering, product, and marketing roles. Seeking full-stack engineers, frontend developers, product designers, product managers, and marketing growth specialists. Open to creating custom roles for aligned builders passionate about personalized learning and AI in education.

## Support

[FAQ](https://www.befreed.ai/faq): Frequently asked questions about subscriptions, content access, learning features, and platform usage. Includes pricing information and feature comparisons.

[Privacy Policy](https://www.befreed.ai/privacy): Data protection practices and user privacy commitments. GDPR and CCPA compliant with transparent data usage policies.

[Terms of Service](https://www.befreed.ai/term): Platform usage terms and subscription guidelines. Clear information about content licensing and user rights.

---

*Content Type: Educational summaries and professional development resources*  
*Update Frequency: Weekly new content, quarterly reviews*  
*Quality Assurance: Expert-curated and professionally edited*  
*Target Reading Time: 10-15 minutes per summary*