A file at the root of your website that tells web crawlers (including AI crawlers) which parts of your site they can or cannot access.
robots.txt is a standard file (at example.com/robots.txt) that controls crawler access to your site. For AEO purposes, robots.txt should explicitly Allow major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Anthropic-AI, Applebot-Extended, etc.) to ensure your content is citation-eligible. Many sites' default robots.txt either blocks AI crawlers (excluding them from AEO) or doesn't explicitly allow them (resulting in inconsistent indexing).
robots.txt is the technical gatekeeper for AEO. Misconfigurations here invisibly block your AEO program. Most teams don't audit robots.txt regularly and discover problems only after pipeline impact.
Lantern's robots.txt explicitly allows 20+ AI crawlers including GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Cohere, Bytespider, Diffbot, and others. This ensures all major AI engines can index Lantern's content for both training and real-time citation.
The terms in this glossary aren't theoretical — they're what Lantern's product calculates and reports every month for B2B SaaS teams. See yours in 7 days. 14-day free trial.
Join Waitlist