Top Sites>Celestia - Fibre Optic Performance for Millisecond Markets | Celestia

# Robots.txt for Celestia.org
# Optimized for AI/LLM crawlers and traditional search engines

# ——— OPENAI ———
# Search (shows webpages as links inside ChatGPT search) - NOT used for model training
User-agent: OAI-SearchBot
Allow: /

# User-driven browsing from ChatGPT and Custom GPTs - Acts after a human click
User-agent: ChatGPT-User
User-agent: ChatGPT-User/2.0
Allow: /

# Model-training crawler - Allow for better representation in AI models
User-agent: GPTBot
Allow: /
Crawl-delay: 2

# ——— ANTHROPIC (Claude) ———
User-agent: anthropic-ai      # bulk model training
Allow: /
Crawl-delay: 2

User-agent: ClaudeBot         # chat citation fetch
User-agent: claude-web        # web-focused crawl
Allow: /

# ——— PERPLEXITY ———
User-agent: PerplexityBot     # index builder
Allow: /

User-agent: Perplexity-User   # human-triggered visit
Allow: /

# ——— GOOGLE (Gemini) ———
User-agent: Google-Extended
Allow: /

User-agent: Googlebot
Allow: /

# ——— MICROSOFT (Bing / Copilot) ———
User-agent: BingBot
Allow: /

# ——— AMAZON ———
User-agent: Amazonbot
Allow: /

# ——— APPLE ———
User-agent: Applebot
User-agent: Applebot-Extended
Allow: /

# ——— META ———
User-agent: FacebookBot
User-agent: meta-externalagent
User-agent: facebookexternalhit
Allow: /

# ——— LINKEDIN ———
User-agent: LinkedInBot
Allow: /

# ——— BYTEDANCE (TikTok) ———
User-agent: Bytespider
Allow: /

# ——— DUCKDUCKGO ———
User-agent: DuckAssistBot
Allow: /

# ——— OTHER AI/RESEARCH CRAWLERS ———
User-agent: AI2Bot
User-agent: CCBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: omgili
User-agent: YouBot
User-agent: MistralAI-User
Allow: /

# ——— DEFAULT FOR ALL OTHER CRAWLERS ———
User-agent: *
Allow: /
Crawl-delay: 1

# ——— SITEMAP LOCATION ———
Sitemap: https://celestia.org/sitemap.xml 

# Celestia - Modular Blockchain Network
> Celestia is the **modular blockchain** powering unstoppable applications with full-stack control. The first network to enable anyone to deploy their own blockchain in minutes.

Celestia provides scalable data availability for rollups, enabling new types of decentralized applications that weren't possible before.

## Core Technology
- [What is Celestia](https://celestia.org/what-is-celestia/): complete beginner's guide to modular blockchains
- [What is Data Availability](https://celestia.org/what-is-da/): data availability layer fundamentals 
- [What is TIA](https://celestia.org/what-is-tia/): Celestia's native staking and fee token
- [Run a Light Node](https://celestia.org/run-a-light-node/): participate in the network

## Learn About Modular Blockchains
- [Modular Blockchains for Beginners](https://celestia.org/learn/beginners/modular-blockchains-for-beginners/): introduction to modular design
- [Modular vs Monolithic Blockchains](https://celestia.org/learn/intermediates/modular-and-monolithic-blockchains/): architecture comparison
- [Benefits of Modular Blockchains](https://celestia.org/learn/intermediates/benefits-of-modular-blockchains/): scalability and sovereignty advantages
- [The Modular Stack](https://celestia.org/learn/intermediates/the-modular-stack/): execution, settlement, consensus, data availability layers

## Rollups and Sovereign Chains  
- [Sovereign Rollups Introduction](https://celestia.org/learn/intermediates/sovereign-rollups-an-introduction/): new blockchain architecture
- [Sovereign Rollups Misconceptions](https://celestia.org/learn/intermediates/sovereign-rollups-misconceptions/): clarifying common myths
- [Settlement in the Modular Stack](https://celestia.org/learn/intermediates/settlement-in-the-modular-stack/): role of settlement layers

## Developer Resources
- [Build on Celestia](https://celestia.org/build/): developer portal and tools
- [Ecosystem](https://celestia.org/ecosystem/): apps and rollups built on Celestia
- [Glossary](https://celestia.org/glossary/): technical terms and definitions

## Community and Company
- [Community](https://celestia.org/community/): global developer and researcher network
- [Careers](https://celestia.org/careers/): join the core team
- [Press](https://celestia.org/press/): media resources and brand assets

## Optional
- [Past Events](https://celestia.org/past-events/): conference talks and workshops
- [Privacy Policy](https://celestia.org/privacy/): data handling practices
- [Terms of Service](https://celestia.org/tos/): platform usage terms

_Last updated: 2025-01-09 · License: MIT · Contact: hello@celestia.org_