# The Food Institute - Your Source for Food Industry News, Data, & Trends > Markdown mirror of DialtoneApp's public top-site detail page for `foodinstitute.com`. URL: https://dialtoneapp.com/top-sites/foodinstitute.com/index.md Canonical HTML: https://dialtoneapp.com/top-sites/foodinstitute.com ## Summary - Domain: `foodinstitute.com` - Website: https://foodinstitute.com - Description: ai readable | score 20 | purchase read only - Label: ai_readable - Payment surface: Not available - Purchase boundary: read_only - Control boundary: unknown - Rank: 165821 ## robots ~~~text # Food Institute - Robots.txt # Updated: 2026-01-11 # Allow major search engines (Google, Bing) full access User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # AI Training Bots - Follow same rules as llms.txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Google-Extended User-agent: GoogleOther User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: FacebookBot User-agent: cohere-ai User-agent: Diffbot User-agent: anthropic-ai Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /wp-content/uploads/ewww/ Disallow: /wp-content/uploads/ewww-3/ Disallow: /wp-json/ Disallow: /ewww/ Disallow: /ewww-3/ Disallow: /cgi-bin/ Disallow: /.well-known/ Disallow: /securefile/ Disallow: /.sucuriquarantine/ Disallow: /reports/2020/ Disallow: /reports/2021/ Disallow: /reports/2022/ Disallow: /reports/2023/ Disallow: /reports/eco/ Disallow: /reports/fir/ Disallow: /reports/du_pdf/ Disallow: /reports/join/ Disallow: /reports/mcith/ Disallow: /food1/ Disallow: /test-site/ Disallow: /feed/ Disallow: /trackback/ Disallow: /xmlrpc.php Disallow: /*? Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.css$ Disallow: /comments/ Disallow: /author/ Disallow: /tag/ Disallow: /page/ # Commercial SEO Bots - Block entirely (also in .htaccess) User-agent: AhrefsBot User-agent: SemrushBot User-agent: SERankingBot User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: SEOkicks User-agent: Barkrowler User-agent: Netvibes User-agent: Amazonbot Disallow: / # Known Rule-Breakers - Block entirely (also in .htaccess) User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Bytespider User-agent: CCBot User-agent: Omgilibot User-agent: webzio-extended User-agent: ImagesiftBot Disallow: / # Sitemap location (optional - WordPress generates this) Sitemap: https://foodinstitute.com/sitemap.xml ~~~ ## llms ~~~text # Food Institute # Leading food industry news and analysis platform # https://foodinstitute.com # Key Resources for AI Indexing ## [Latest News & Articles](https://foodinstitute.com/) - Breaking food industry news, market analysis, and trends ## [Industry Reports](https://foodinstitute.com/reports/) - Comprehensive market intelligence and research ## [Daily Updates](https://foodinstitute.com/reports/dailyupdate/) - Daily food industry briefings and market insights ## [About Us](https://foodinstitute.com/about/) - Learn about Food Institute's mission and services # Company Information - Company: Food Institute - Focus: Food industry news, analysis, and market intelligence - Content: Industry reports, daily briefings, market trends # AI Crawling Policy - OPTIMIZED FOR BANDWIDTH User-agent: * # WordPress Core - BLOCK (No value for AI, wastes bandwidth) Disallow: /wp-admin/ # WordPress admin Disallow: /wp-includes/ # WordPress core files Disallow: /wp-content/plugins/ # Plugin files Disallow: /wp-content/themes/ # Theme files Disallow: /wp-content/uploads/ewww/ # Image optimization cache Disallow: /wp-content/uploads/ewww-3/ # Image optimization cache Disallow: /wp-json/ # REST API endpoints # Image Optimization Caches - BLOCK Disallow: /ewww/ # EWWW cache directory Disallow: /ewww-3/ # EWWW cache directory (duplicate) # Security & System - BLOCK Disallow: /cgi-bin/ # CGI scripts Disallow: /.well-known/ # SSL validation files Disallow: /securefile/ # Security files Disallow: /.sucuriquarantine/ # Sucuri quarantine # Archive Optimization - BLOCK OLD CONTENT Disallow: /reports/2020/ # Outdated reports (5+ years old) Disallow: /reports/2021/ # Outdated reports (4+ years old) Disallow: /reports/2022/ # Outdated reports (3+ years old) Disallow: /reports/2023/ # Older reports (2+ years old) # Test/Cache Directories - BLOCK Disallow: /reports/eco/ # Old/test content Disallow: /reports/fir/ # Old/test content Disallow: /reports/du_pdf/ # PDF cache directory Disallow: /reports/join/ # Join/signup forms Disallow: /reports/mcith/ # MCITH content Disallow: /test-site/ # test site # Subdomain - BLOCK (staging/test sites) Disallow: /food1/ # food1-co subdomain files Disallow: /test-site/ # test site # Functional Pages - BLOCK (No content value) Disallow: /feed/ # RSS feeds Disallow: /trackback/ # Trackback endpoints Disallow: /xmlrpc.php # XML-RPC Disallow: /*? # URLs with query parameters (search, etc) Disallow: /*.php # PHP files Disallow: /*.js # JavaScript files Disallow: /*.css # Stylesheets Disallow: /*.inc # Include files # User-Generated/Dynamic Content - BLOCK Disallow: /comments/ # Comment pages Disallow: /author/ # Author archives Disallow: /tag/ # Tag pages Disallow: /page/ # Pagination # Everything else is allowed (homepage, /about/, /reports/2024/, /reports/2025/, etc.) # Training Guidelines Training-Data: allowed Commercial-Use: allowed-with-attribution Attribution: required Modification: allowed Distribution: allowed-with-source-link Data-Collection-Consent: implicit # Explanation We allow AI training on our current news articles and industry reports (2024-2025) to help spread food industry knowledge. Older archives and WordPress system files are excluded to reduce unnecessary bandwidth consumption. # Metadata Crawl-delay: 2 Categories: food-industry, news, market-intelligence, reports, analysis Last-modified: 2026-01-11 Version: 1.0 Content-Focus: food-industry-news, market-analysis, daily-updates # Custom Directives Research: encouraged Educational-use: encouraged Commercial-training: allowed-with-attribution User-generated-content: excluded System-files: excluded Archive-content: exclude-before-2024 # Preferred Indexing Priority-content: /reports/dailyupdate/, /reports/2024/, /reports/2025/, /reports/marketinfo/ Update-frequency: daily ~~~ ## llms-full Not found.