AI-Ready Robots.txt Generator
Build explicit rules for search indexing, user-requested retrieval, and model-training access. Each provider publishes different controls, so this generator keeps those purposes separate.
Basic Settings
AI crawler and data-use controls
Choose the access policy that fits each agent. Search, user-requested, and training agents can have different rules from the same provider.
Search Engine Crawlers
robots.txt
# robots.txt policy for search, AI retrieval, and model training # Generated: 2026-09-13 # === Default rules for all crawlers === User-agent: * Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ # === AI search, retrieval, and data-use controls: ALLOWED === User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Claude-SearchBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Claude-User Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: PerplexityBot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Google-Extended Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Applebot Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ User-agent: Meta-WebIndexer Allow: / Disallow: /admin Disallow: /api/ Disallow: /private/ # === AI search, retrieval, and data-use controls: BLOCKED === User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / # === Sitemap === Sitemap: https://example.com/sitemap.xml
Check your current robots.txt
Run a Technical Audit to test crawler access and measure the page's eight AI Readiness checks.
Scan Your Website FreeWhat does an AI robots.txt policy control?
A robots.txt rule controls whether a named agent may request a path. It does not promise that an answer engine will discover, index, mention, or cite the page. The useful decision is which purpose you want to allow for each provider.
| Purpose | Representative controls | What the rule changes |
|---|---|---|
| Search and indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Googlebot, Applebot | Whether the provider can collect pages for search and answer retrieval. |
| User-requested retrieval | ChatGPT-User, Claude-User | Whether an assistant can fetch a page while responding to a user. |
| Training and data use | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot | Whether content may enter model, grounding, or dataset pipelines, subject to each provider's policy. |
Keep private paths private
Repeat sensitive paths under each allowed agent. Robots.txt is a public crawl policy, so authentication remains the real protection for private content.
Separate search from training
OpenAI, Anthropic, and Apple publish separate controls. That lets a site remain eligible for search or user-requested fetches while declining model-training use.
Verify beyond the file
Check server responses, WAF rules, and request logs after publishing. An allow rule cannot override a CDN challenge, authentication wall, or blocked IP range.
Learn more about robots.txt and AI crawlers
How do I use the robots.txt generator?
Make one policy decision at a time. Search visibility, live retrieval, and model training are separate jobs, even when the same company runs all three agents.
Step 1
Set the default crawl rule
Allowing all crawlers by default keeps public pages discoverable. A site that must deny unknown agents can start with a default block, then explicitly allow the named search agents it trusts.
Step 2
Choose rules by purpose
Keep search and user-requested agents enabled when discoverability is the goal. Decide separately whether training agents and data-use tokens may use the same public content.
Step 3
Protect paths and add the sitemap
List admin, API, staging, cart, or other non-public paths once per line. Add the absolute sitemap URL so compliant crawlers can find the public pages you do want indexed.
Step 4
Publish and verify the response
Save the output at the site root as
/robots.txt. Confirm that it returns HTTP 200 without a login, redirect loop, or bot challenge, then inspect server logs for the agents you allowed.
Crawl delay is a non-standard directive. Anthropic and Common Crawl document support for it, while other providers may ignore it. Rate limits at the CDN or origin remain the reliable control when a provider does not honor that field. Treat every generated directive as a published request, then confirm the real behavior in access logs. A user-agent string alone also does not prove that a request came from the named provider. Verify published IP ranges or reverse DNS when the provider offers them.
Frequently asked questions
Which crawler rules matter most for AI search visibility?
Start with the search and indexing agents: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude search, PerplexityBot for Perplexity search, Meta-WebIndexer for Meta AI search, Googlebot for Google Search and AI Overviews, and Applebot for Apple search. User-requested agents such as ChatGPT-User and Claude-User are separate from training crawlers.
Can I allow AI search while blocking model training?
Yes, when the provider publishes separate controls. You can allow OAI-SearchBot while blocking GPTBot, allow Claude-SearchBot and Claude-User while blocking ClaudeBot, and allow Applebot while blocking Applebot-Extended. Google-Extended controls Gemini model training and grounding in Gemini Apps, but it does not control inclusion in Google Search.
Does allowing an AI crawler guarantee a citation?
No. An allow rule makes a URL eligible for that crawler to request. It does not guarantee discovery, indexing, selection, a brand mention, or a citation. Links, crawlable HTML, clear answers, authority, and the provider's retrieval system still determine whether a page appears in an answer.
Do all AI user agents obey robots.txt?
No. Provider behavior differs by agent. Perplexity says Perplexity-User generally ignores robots.txt because the fetch is requested by a user, so this generator does not present it as a controllable rule. Use provider documentation and server logs to verify the behavior that matters to your site.
Provider sources
Crawler names and purposes were checked against provider documentation on September 9, 2026. Recheck these sources before making a high-stakes access decision because providers can change their agents and policies.
- OpenAI publisher guidance ↗
- Anthropic crawler documentation ↗
- Perplexity crawler documentation ↗
- Google crawler documentation ↗
- Applebot documentation ↗
- Meta web crawler documentation ↗
- Common Crawl CCBot documentation ↗
After publishing, compare the file with the robots.txt tester and check live access across 28 named agents with the AI crawler access checker.
Related reading
Go beyond one check
Check your whole site, not just robots.txt generation
Run a complete Technical Audit for SEO, AI Readiness, performance, security, and accessibility. No signup required.
Free technical audit. All issues, all categories. No signup required.