Robots.txt Tester & Analyzer
Analyze any website's robots.txt file. See which crawlers are allowed or blocked, check AI bot access, and find SEO issues.
What is robots.txt?
The robots.txt file tells search engines and other crawlers which parts of your website they can and can't access. It lives at the root of your domain (e.g., example.com/robots.txt) and uses a simple text format with User-agent, Allow, and Disallow directives. Getting it wrong can prevent Google from indexing your site or block AI crawlers from citing your content.
Why Test Your robots.txt?
SEO Impact
A misconfigured robots.txt can accidentally block Googlebot from your entire site, causing all pages to be deindexed. This is one of the most common SEO disasters.
AI Search Visibility
AI crawlers like GPTBot (ChatGPT), ClaudeBot, and PerplexityBot check robots.txt. Blocking them means your content won't appear in AI-generated answers.
Crawl Budget
For large sites, robots.txt controls how search engines spend their crawl budget. Blocking low-value pages (admin, search results) helps prioritize important content.
Security
robots.txt can inadvertently reveal hidden paths (e.g., /admin, /staging). Attackers scan robots.txt for directories to probe. Review what you're disclosing.
Frequently Asked Questions
Should I block AI crawlers in robots.txt?
It depends on your goals. If you want your content to appear in ChatGPT, Perplexity, and Google AI Overviews, allow AI crawlers. If you're concerned about AI training on your content, you can block specific bots while keeping search engines allowed. Use our tool to see which bots you're currently blocking.
Does robots.txt prevent indexing?
No. robots.txt prevents crawling, not indexing. Google can still index a URL it finds in links or sitemaps, even if robots.txt blocks crawling. To prevent indexing, use a noindex meta tag or X-Robots-Tag header instead.
What happens if I don't have a robots.txt?
Without a robots.txt, all crawlers are free to access all pages. This is fine for most small sites. But for larger sites, you may want to block admin pages, search result pages, or duplicate content paths to optimize crawl budget.
How often should I check my robots.txt?
Check whenever you make structural changes to your site (new sections, redesigns, migrations). Also check after deploying new features — it's common for frameworks to accidentally add overly restrictive robots.txt rules.