What is llms.txt? How This New Standard Helps AI Discover Your Website
AI search is reshaping how people find businesses. A new proposed standard called llms.txt gives AI models a structured brief about your entire website — helping them understand, cite, and recommend you.
Search is splitting in two. On one side, traditional search engines still drive billions of clicks through blue links. On the other, AI-powered answer engines — ChatGPT, Perplexity, Google AI Overviews, Claude — are synthesizing answers from across the web, often without a click at all. According to Gartner, 25% of search volume will shift to AI engines by 2026. That shift is already underway, and it raises a critical question: how do AI models decide which websites to trust, cite, and recommend?
Enter llms.txt. It's a proposed standard — similar in spirit to robots.txt — that gives AI language models a structured summary of your website's content, purpose, and architecture. Think of it as a cover letter for your site, written specifically for an AI audience.
This guide covers what llms.txt is, why it matters for your GEO and AEO strategy, how to create one, and the mistakes you should avoid.
Quick Definition
llms.txt is a proposed standard file placed at a website's root that provides AI language models with a structured description of the site's content, key pages, products, and purpose — enabling better AI understanding and citation.
The Problem: AI Engines Struggle to Understand Complex Websites
When you ask ChatGPT or Perplexity a question, these AI models need to figure out which sources to trust and cite. They crawl pages, parse content, and try to build an understanding of what each website offers. But modern websites are complex — dynamic navigation, JavaScript-rendered content, sprawling page hierarchies, and marketing copy that obscures the actual product.
The result? AI models often miss key pages, misunderstand what a company does, or simply skip over websites that don't make their content easy to parse. A Wynter study from 2026 found that 84% of B2B CMOs now use AI and LLMs for vendor discovery. If AI can't understand your site, you're invisible to a growing share of your buyers.
Traditional SEO signals — backlinks, keyword density, domain authority — help with Google rankings, but they don't help an AI model understand your site's structure. That's the gap llms.txt is designed to fill.
llms.txt vs robots.txt: Different Jobs, Complementary Roles
If you're familiar with robots.txt, you might wonder how llms.txt is different. The short answer: they solve different problems entirely.
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Purpose | Access control — which pages crawlers can visit | Content description — what the site is about |
| Audience | Search engine crawlers (Googlebot, GPTBot) | AI language models parsing site context |
| Answers the question | “Can you access this page?” | “What is this site about?” |
| Format | Directive-based (Allow/Disallow rules) | Markdown-formatted descriptive content |
| Adoption | Universal standard since 1994 | Emerging standard, early adoption phase |
The key takeaway: robots.txt is a bouncer at the door. llms.txt is a tour guide once you're inside. You need both. Allowing AI crawlers to access your site (via robots.txt) is step one. Helping them understand what they're looking at (via llms.txt) is step two.
What to Include in Your llms.txt File
A good llms.txt file gives an AI model everything it needs to understand your business at a glance. Think of it as answering the question: “If an AI could only read one file about our company, what should it say?”
Here are the key sections to include:
1. Company Description
A concise, factual summary of what your company does. Avoid marketing jargon. AI models perform best with clear, declarative statements. State who you are, what you offer, and who your target audience is.
2. Key Pages and Their Purpose
List your most important pages with a one-line description of each. This tells AI models which URLs to prioritize when answering questions about your product or service. Include your homepage, pricing page, documentation, and core product pages.
3. Product and Service Information
Describe your products or services in plain language. Include what problems they solve, who they're for, and what makes them different. This is what AI models reference when users ask “What tools can help me with X?”
4. Team and Authority Signals
If relevant, mention key team members, their credentials, and the company's expertise. AI models weigh authority signals when deciding what to cite. A company with recognized experts in a field is more likely to be recommended.
5. Pricing Overview
If your pricing is public, include a brief summary. AI models frequently answer “How much does X cost?” queries. Having pricing in your llms.txt makes it easy for AI to provide accurate answers and link back to your pricing page.
6. Documentation and Resource Links
Point to your API docs, help center, tutorials, and other resources. Developers researching tools through AI assistants like Claude Code or Cursor rely on these references. Making them discoverable increases your technical audience reach.
How to Create and Deploy Your llms.txt File
Implementing llms.txt is straightforward. There's no special tooling required — just a text file with markdown formatting, hosted at your site's root.
Step 1: Create the File
Create a new file called llms.txt in your website's public or root directory. The file uses markdown formatting, which makes it both human-readable and AI-parseable.
Step 2: Structure Your Content
Here's an example structure for a SaaS company:
# Company Name > One-line description of what your company does. ## About Brief company overview. What you do, who you serve, and what makes you different. 2-3 sentences max. ## Key Pages - [Homepage](https://yoursite.com) - Main landing page - [Pricing](https://yoursite.com/pricing) - Plans and pricing - [Documentation](https://yoursite.com/docs) - Technical docs - [Blog](https://yoursite.com/blog) - Industry insights ## Products / Services - **Product A**: Description of what it does and who it's for - **Product B**: Description of what it does and who it's for ## Team - **Jane Smith, CEO** - Background and expertise - **John Doe, CTO** - Background and expertise ## Contact - Website: https://yoursite.com - Email: hello@yoursite.com - Support: https://yoursite.com/support
Step 3: Host at /llms.txt
Place the file so it's accessible at https://yoursite.com/llms.txt. On most platforms:
- Next.js / Vercel: Place the file in your
public/directory - WordPress: Upload to your site's root directory via FTP or file manager
- Static sites: Include in your build output directory alongside
robots.txtandsitemap.xml - Squarespace / Wix: Use the custom file or code injection feature to serve the file at the root path
Step 4: Validate and Test
After deployment, verify your file is accessible by visiting yoursite.com/llms.txt in a browser. Check that:
- The file loads with a 200 status code (not a redirect or 404)
- Markdown formatting renders correctly when viewed as plain text
- All links in the file resolve to live pages
- The content accurately reflects your current product offering
How llms.txt Fits Into the GEO/AEO Flywheel
Generative Engine Optimization (GEO) is a multi-layered discipline. There's no single tactic that makes your site AI-visible. Instead, it's a flywheel of complementary signals that work together. Here's where llms.txt fits:
- Access layer (robots.txt): AI crawlers can reach your site — configure your robots.txt for AI crawlers
- Site-level context (llms.txt): AI models understand your entire site at a glance — that's what we're covering here
- Page-level structure (schema markup): Individual pages are machine-readable — schema markup for AI search
- Content quality: Your content is authoritative, current, and citation-worthy — content updated within 30 days gets 3.2x more AI citations
- Monitoring and iteration: You track AI citations and adjust — the complete AI SEO checklist for 2026
Without llms.txt, you're relying on AI models to piece together your site's purpose from scattered pages. With it, you're handing them a clear, structured brief. It's the difference between hoping AI understands you and making sure it does.
Which AI Engines Currently Support llms.txt?
Honesty is important here: llms.txt is still an emerging standard. It doesn't have the universal adoption that robots.txt enjoys after three decades. But early momentum is building:
- LLM-powered developer tools (Cursor, Claude Code, Windsurf) already look for project-level context files. The pattern of providing structured context to AI is well-established in developer workflows.
- AI search engines like Perplexity and ChatGPT browse the web in real time. A well-formatted
llms.txtfile provides a clean, parseable summary that these models can leverage during content synthesis. - Google AI Overviews pull from structured and semi-structured content. While Google hasn't explicitly endorsed
llms.txt, the principle of providing clear, structured site descriptions aligns with how AI Overviews select sources. - Enterprise AI platforms that use retrieval-augmented generation (RAG) benefit from any structured context file, including
llms.txt, when processing website data.
The strategic argument is simple: when llms.txt becomes widely supported, sites that already have one will have a head start. The cost of creating the file is minimal. The downside of not having one when adoption accelerates is significant.
Best Practices for llms.txt
Follow these guidelines to maximize the value of your llms.txt file:
- Be factual, not promotional. AI models are trained to detect and discount marketing fluff. Write in clear, declarative sentences. “We help B2B SaaS companies track AI search visibility” is better than “The world's most innovative AI-powered revolutionary platform.”
- Keep it current. Update your
llms.txtwhenever you launch new products, change pricing, or restructure your site. Stale information undermines trust. Remember: content updated within 30 days gets 3.2x more AI citations. - Use markdown formatting consistently. Headings, bullet points, and links should follow standard markdown syntax. This ensures AI models can parse the structure reliably.
- Link to canonical URLs. Every URL you reference should be the canonical version of that page. Avoid linking to redirect URLs, tracking-parameterized URLs, or URLs that resolve to a different page.
- Match your actual content. If your
llms.txtdescribes a feature your site doesn't actually offer, AI models will eventually learn to discount your file. Accuracy builds trust. - Keep it concise. Aim for 500–1,500 words. Long enough to be comprehensive, short enough to be parsed quickly. AI models have context windows, and a bloated file risks having important information truncated.
- Include your unique value proposition. What differentiates you from competitors? AI models compare multiple sources. If you don't clearly state what makes you different, they have no reason to recommend you over alternatives.
Common Mistakes to Avoid
Even well-intentioned llms.txt implementations fall short. Here are the most common pitfalls:
- Treating it as a marketing brochure. Superlatives, buzzwords, and vague claims make your file less useful to AI models. “Industry-leading, best-in-class solution” tells an AI nothing. “Website audit tool that checks 47 SEO and GEO factors” tells it exactly what you do.
- Including every page on your site. This isn't a sitemap. Focus on 10–20 key pages that represent your core offering. AI models need a curated overview, not an exhaustive index.
- Setting it and forgetting it. An
llms.txtthat references deprecated products or outdated pricing erodes AI trust in your site. Build a quarterly review into your content calendar. - Using non-standard formatting. Stick to standard markdown. Custom HTML, embedded scripts, or proprietary formatting won't be parsed correctly by most AI models.
- Confusing it with robots.txt. Adding access control directives to
llms.txtmisses the point. This file is about providing context, not restricting access. Keep access rules inrobots.txtwhere they belong. - Forgetting to verify the file is accessible. A
llms.txtthat returns a 404, redirects to your homepage, or requires authentication is useless. Test the URL after deployment.
How Foglift Checks Your llms.txt
Foglift's Website Audit already checks for llms.txt as part of its GEO Readiness scoring. Here's what the audit evaluates:
- Presence: Does your site have an
llms.txtfile at the root? If not, you'll see a recommendation to add one. - Accessibility: Does the file return a 200 status, or does it redirect or fail?
- Structure: Is the file properly formatted with markdown headings, descriptions, and links?
- Completeness: Does it cover the essential sections — company description, key pages, products, and contact information?
- Freshness: Are the links current and do they resolve to live pages?
The llms.txt check is one factor in your overall GEO score, alongside AI crawler access, structured data, content quality, and other signals. Together, these factors determine how ready your site is for AI-driven discovery.
The Bigger Picture: AI Readiness Is a Competitive Advantage
Every standard that helps AI understand the web follows the same adoption curve. Early adopters gain an outsized advantage, and laggards scramble to catch up. robots.txt went from a niche webmaster convention to a universal standard. Schema markup went from a curiosity to a ranking factor. llms.txt is on the same trajectory.
The companies that will dominate AI search visibility in the next 12 months are the ones building their AI readiness stack now. That stack includes:
- Properly configured
robots.txtallowing AI crawlers - A well-structured
llms.txtproviding site-level context - Comprehensive schema markup on every key page
- Fresh, authoritative, citation-friendly content
- Regular monitoring of AI citations and recommendations
Miss any one of these layers and you leave a gap that competitors will fill.
Frequently Asked Questions
What is llms.txt?
A proposed standard file that websites place at their root to help AI language models understand the site's content, structure, and purpose. It uses markdown formatting to provide a structured summary — covering company description, key pages, products, team, and resources — so AI engines can quickly grasp what a site offers without crawling every page.
Is llms.txt the same as robots.txt?
No. robots.txt controls access — it tells crawlers which pages they can and cannot visit. llms.txt provides context — it describes what the site is about, highlights key pages, and explains how to use the information. They serve complementary but fundamentally different purposes.
Do I need llms.txt if I already have structured data?
They serve different purposes. Schema markup helps AI understand individual pages — their type, author, topic, and relationships. llms.txt helps AI understand your entire site at a glance. Think of structured data as page-level detail and llms.txt as a site-level brief. Using both gives AI the most complete picture.
Which AI engines support llms.txt?
The standard is still emerging, and no major AI engine has formally announced llms.txt support as a ranking signal. However, early adopters include LLM-powered developer tools and AI search engines that browse the web in real time. Regardless of formal support, having a well-structured llms.txt improves your overall AI readiness and positions your site for when adoption accelerates.
How does llms.txt affect my GEO score?
A well-structured llms.txt contributes to your site's AI readiness, which is a component of your overall GEO score. Foglift's Website Audit evaluates your llms.txt for presence, structure, completeness, and freshness as part of its GEO scoring methodology.
Check Your AI Readiness Score
Foglift audits your website for llms.txt, AI crawler access, structured data, and 40+ other GEO factors. Free, instant results.
Free Website Audit