llms.txt for AI Search Visibility

llms.txt is a proposed convention for giving language models and agents a concise map of a website. You publish a Markdown file at /llms.txt, describe the site in plain language, and link to the pages that carry the most useful facts. The file is cheap to ship and easy to maintain. Its effect on AI search visibility has not been proven.

That evidence boundary matters. The original proposal describes an inference-time aid for models facing limited context windows. It does not define a ranking system. As of July 13, 2026, the public crawler guidance from OpenAI and Anthropic documents robots.txt controls for training, search, and user-directed retrieval. Neither provider promises that an llms.txt file will improve citations or rankings.

What the llms.txt Proposal Actually Specifies

Jeremy Howard published the proposal on September 3, 2024. It asks sites to expose a Markdown file at the root path. The goal is to give an LLM a short project summary and a curated set of links without forcing it to parse a full site, its navigation, and its JavaScript.

The proposed order is deliberately simple:

1. An H1 containing the project or site name. This is the only required field.
2. An optional blockquote with the short description.
3. Optional paragraphs or lists with facts and interpretation guidance.
4. H2 sections containing Markdown links with brief descriptions.
5. An optional ## Optional section for links that can be skipped when context is limited.

The proposal also suggests clean Markdown versions of important pages, often exposed with a .md suffix. That extension is optional. A useful first release can link to normal canonical HTML pages when those pages are concise, server-rendered, and readable without authentication.

llms.txt, robots.txt, and sitemap.xml

OpenAI separates OAI-SearchBot for search, GPTBotfor potential training use, and ChatGPT-User for user-triggered visits. Anthropic documents the same functional split through Claude-SearchBot, ClaudeBot, and Claude-User. Configure those access rules in robots.txt. Use llms.txt to explain which public pages are most useful after a system or person reaches the site.

A B2B SaaS llms.txt Template

A good file answers four questions quickly: What does the product do? Who is it for? Which pages contain current commercial facts? Where can an agent find implementation details? This template stays within the proposed structure and avoids promotional claims that the linked pages cannot prove.

# Acme Analytics

> Acme Analytics helps B2B SaaS teams connect product usage to revenue. It serves product, growth, and RevOps teams.

Key facts:
- Product: product analytics and revenue attribution
- Customers: B2B SaaS companies
- Pricing: public plans with a 14-day trial
- Canonical domain: https://www.example.com

## Product

- [Product overview](https://www.example.com/product): Core capabilities and supported use cases
- [Pricing](https://www.example.com/pricing): Current plans, limits, and billing terms
- [Security](https://www.example.com/security): Data handling, compliance, and subprocessors

## Documentation

- [Quickstart](https://www.example.com/docs/quickstart): Install the SDK and send the first event
- [API reference](https://www.example.com/docs/api): Authentication, endpoints, and response formats
- [Integrations](https://www.example.com/integrations): Supported data sources and destinations

## Optional

- [Customer stories](https://www.example.com/customers): Verified implementation examples
- [Company](https://www.example.com/about): Organization and leadership information

What to Include and What to Leave Out

Choose canonical pages that resolve without a login and stay current. For most SaaS sites, the useful core is the product overview, pricing, security, quickstart, API reference, and integrations directory. Add status or trust pages when buyers and agents regularly ask operational questions.

• Use one precise description. Match the entity statement used on your homepage and Organization schema.
• Use absolute URLs. They remain unambiguous when the file is copied into another context.
• Describe each link. Explain what facts or tasks the destination supports.
• Keep pricing factual. Link to the current pricing page instead of duplicating a large plan matrix that will drift.
• Keep private surfaces out. Dashboard URLs, customer data, internal documentation, and staging hosts do not belong in the file.
• Remove dead or superseded pages. A small accurate index is more useful than a complete archive.

How to Publish llms.txt

A static site can place the file in its public directory. A Next.js App Router project can use public/llms.txt or a route handler when the contents are generated. Either approach should return public Markdown or plain text at the exact root URL.

1. Draft the file from canonical public pages.
2. Publish it at https://yourdomain.com/llms.txt.
3. Confirm an unauthenticated request returns HTTP 200.
4. Check that redirects land on canonical HTTPS URLs.
5. Run every linked URL through a broken-link check.
6. Add file maintenance to the release checklist for pricing, docs, and product changes.

How to Measure Whether It Helps

Presence is easy to test. Search impact requires a controlled measurement plan because model answers vary across runs and engines. Record a baseline before publication, keep the prompts stable, and resist attributing a citation change to the file when other pages or authority signals changed during the same window.

1. Select 10 to 20 factual prompts about the product, pricing, integrations, and category.
2. Run each prompt across the same engines and record mention, citation, and factual accuracy.
3. Publish the file without changing the linked pages during the test window.
4. Review server logs for requests to /llms.txt and the linked URLs.
5. Repeat the prompt set for at least two weeks and compare rates by engine.

A successful first implementation can be operational even when visibility stays flat. The file gives developers and agents a stable site index, exposes stale claims during review, and creates a clear maintenance surface. Any ranking or citation lift should remain a measured result from your own prompt set.

Frequently Asked Questions

Primary Sources

• llms.txt proposal and format, Jeremy Howard, September 2024
• OpenAI crawler documentation for search, training, and user requests
• Anthropic crawler documentation, updated April 7, 2026

Fundamentals: Learn about GEO (Generative Engine Optimization) and AEO (Answer Engine Optimization) (the two frameworks for optimizing your content for AI search engines).