AI-Ready Robots.txt Generator
Generate a robots.txt optimized for both search engines and AI crawlers. Control which AI models — ChatGPT, Claude, Perplexity, Gemini — can access your content.
Basic Settings
AI Crawlers
6/12 allowedChoose which AI bots can crawl your site. Allowing more AI crawlers increases your chances of appearing in AI-generated answers.
Search Engine Crawlers
robots.txt
# robots.txt generated by Foglift.io # AI-optimized for both search engines and AI crawlers # Generated: 2026-03-16 # === Default rules for all crawlers === User-agent: * Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ # === AI crawlers: ALLOWED === # These AI bots can access your content for AI-generated answers User-agent: GPTBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Google-Extended Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: ClaudeBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ # === AI crawlers: BLOCKED === # These AI bots cannot access your content User-agent: Meta-ExternalAgent Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / # === Sitemap === Sitemap: https://example.com/sitemap.xml
Check your current robots.txt
Scan your website with Foglift to see if AI crawlers can actually access your content, plus get your full SEO + GEO score.
Scan Your Website FreeWhy AI Crawlers Matter for Your robots.txt
AI Search is Growing
Over 40% of Gen Z uses AI chatbots instead of Google. If AI crawlers can't access your content, you're invisible to this audience.
Control, Don't Block
Many sites accidentally block AI crawlers with generic Disallow rules. Our generator ensures you allow the right bots while keeping private paths protected.
GEO Optimization
robots.txt is the first thing AI crawlers check. It's the foundation of your Generative Engine Optimization (GEO) strategy.
Learn more about robots.txt and AI crawlers