Top Sites>Ddeepdyve.com

# DeepDyve robots.txt
# Updated: 2025-12-15
# Sitemap architecture follows /sitemap-spec.md

# ==================================================
# Default Crawl Rules (All Bots)
# ==================================================
User-agent: *
Disallow: /cgi-bin/
Disallow: /openurl
Disallow: /search
Disallow: /browse-wr/
Disallow: /enterprise-free-trial
Disallow: /rental-link
Disallow: /timescited

Crawl-delay: 5

# ==================================================
# Sitemap Index Reference
# ==================================================
Sitemap: https://www.deepdyve.com/sitemaps/sitemap_index.xml

# ==================================================
# Googlebot-Specific Rules
# ==================================================
User-agent: Googlebot
Disallow: /assets/images/doccover.png
Disallow: /cgi-bin/
Disallow: /openurl
Disallow: /search
Disallow: /browse-wr/
Disallow: /enterprise-free-trial
Disallow: /rental-link
Disallow: /timescited

# ==================================================
# LLM Crawler Permissions
# Per sitemap-spec.md section 8.1
# ==================================================

# OpenAI GPT Crawler
User-agent: GPTBot
Allow: /

# Google Extended (Bard/Gemini training)
User-agent: Google-Extended
Allow: /

# Anthropic Claude Crawler
User-agent: Claude-Web
Allow: /

# ==================================================
# Additional LLM Crawlers (Optional)
# ==================================================

# Common Crawl (used by many AI models)
#User-agent: CCBot
#Allow: /

# Meta AI (Facebook/Instagram AI)
#User-agent: FacebookBot
#Allow: /

# Perplexity AI
#User-agent: PerplexityBot
#Allow: /

# DeepDyve

> DeepDyve is a research platform providing access to over 150 million scholarly articles across thousands of peer-reviewed journals.

## About

DeepDyve offers researchers, students, and professionals unlimited online reading and document rental for academic papers across all disciplines. The platform provides article metadata, abstracts, DOI resolution, and full-text access through subscription.

## Sitemaps

- [Sitemap Index](https://www.deepdyve.com/sitemaps/sitemap_index.xml): Master index for all DeepDyve sitemaps
- [Interim Crawl Sitemaps](https://www.deepdyve.com/sitemaps/interim/): Sitemaps generated from comprehensive site crawls
- [Top Pages Sitemaps](https://www.deepdyve.com/sitemaps/interim/top/): High-value pages from Google Analytics and Search Console data

## Content

- Research articles include structured metadata following ScholarlyArticle schema
- Each article page contains: title, authors, abstract, DOI, journal information, publication date
- Content is optimized for academic search and citation indexing

## Key Features

- Access to peer-reviewed research across all academic disciplines
- Real-time article search and discovery
- Journal browsing and filtering
- DOI-based article resolution
- Citation tracking and references

## Contact

- Website: https://www.deepdyve.com
- Help Center: https://help.deepdyve.com