AI Crawler Fetches and AI Referrals Measure Different Things
A 28-day sample across 16 sites recorded 212,012 AI crawler fetches and 1,011 recognized AI referrals. Fetch volume did not provide a reliable substitute for human visit measurement.
Methodology
Foglift aggregated requests matching its documented crawler user-agent catalog and recognized AI referral visits from 16 anonymized sites with active Sensor data. The fixed 28-day window ran from September 13 through October 10, 2026. Sites contributed between 5 and 28 observed days because installations started on different dates, so correlation estimates use per-observed-day rates. Crawler purpose follows Foglift's versioned agent catalog; referrals are classified from recognized AI-engine referrer hosts. The report publishes only cohort aggregates, not workspace IDs, domains, paths, or site-level rows. Pearson and Spearman coefficients are descriptive for this small, skewed sample and do not establish causality.
A fetch is evidence of access, not evidence of a visit
AI crawler fetches and AI referral visits were measurably different signals in this cohort. Across 16 anonymized sites, the daily rate of all crawler fetches had a Pearson correlation of 0.04 with the daily rate of recognized AI referrals. The live user-fetch rate had a correlation of -0.09.
The rank correlations were higher, at 0.43 for all crawler fetches and 0.36 for live user fetches. That means sites with more crawler activity often ranked somewhat higher on referrals, but the request counts did not scale with human visits. A fetch can show that an engine reached a page. It cannot show that a person clicked through.
Anonymized Sensor installations with crawler activity
Recorded across 311 observed site-days
Recognized visits from five AI products
12,260 of all recorded crawler requests
Most crawler traffic served training or indexing
Training and indexing accounted for 87.1% of recorded crawler traffic. Live user fetches accounted for 5.8%. Grouping these requests into one bot-traffic total hides the question each request can answer.
| Crawler purpose | Requests | Share | What it indicates |
|---|---|---|---|
| Training | 105,703 | 49.9% | Potential model-training collection |
| Indexing | 78,850 | 37.2% | Search discovery and index maintenance |
| Mixed | 15,173 | 7.2% | Agents with more than one documented role |
| Live user fetch | 12,260 | 5.8% | A request associated with a user action |
| Owner requested | 26 | <0.1% | Site-owner initiated verification |
Provider documentation supports the purpose split. OpenAI separates OAI-SearchBot, GPTBot, and ChatGPT-User. Anthropic separates Claude-SearchBot, ClaudeBot, and Claude-User. Perplexity separates PerplexityBot from Perplexity-User. A single engine can therefore generate several request classes with different implications.
User fetches and referrals did not always co-occur
Eleven sites recorded at least one live user fetch, and eleven recorded at least one recognized AI referral. Those were not the same eleven sites. Two sites had a live user fetch but no referral. Two others had a referral but no live user fetch during the window.
| Live user fetch | Recognized referral | Sites | Cohort share |
|---|---|---|---|
| Yes | Yes | 9 | 56.3% |
| Yes | No | 2 | 12.5% |
| No | Yes | 2 | 12.5% |
| No | No | 3 | 18.8% |
Daily rates show why volume is a weak proxy
Installations entered the cohort on different dates, so the comparison uses requests and referrals per observed day. Pearson measures whether volumes scale linearly. Spearman measures whether sites tend to retain the same rank ordering. Neither coefficient turns a fetch into a visit.
| Crawler signal | Pearson r | Spearman rho | Interpretation |
|---|---|---|---|
| All crawler fetches | 0.04 | 0.43 | Descriptive association, not a conversion rate |
| Training fetches | 0.02 | 0.31 | Descriptive association, not a conversion rate |
| Indexing fetches | 0.12 | 0.50 | Descriptive association, not a conversion rate |
| Live user fetches | -0.09 | 0.36 | Descriptive association, not a conversion rate |
The sample is small and referral traffic is concentrated. One site produced 72.5% of the cohort's referrals. That skew is why this report publishes both linear and rank correlation, and why it does not claim a universal crawler-to-referral rate.
ChatGPT supplied most recognized referrals
ChatGPT accounted for 934 of 1,011 recognized referrals in this frozen cohort. This is a traffic distribution, not a typical-site benchmark. The same high-volume site that creates the concentration caveat above also drives much of the engine split.
| Referrer engine | Visits | Cohort share |
|---|---|---|
| ChatGPT | 934 | 92.4% |
| Gemini | 42 | 4.2% |
| Perplexity | 26 | 2.6% |
| Claude | 8 | 0.8% |
| Copilot | 1 | 0.1% |
Measure the path as separate evidence layers
A useful AI-search measurement stack does not collapse access, selection, and traffic into one score. Read the layers in order:
- Training and indexing fetches show that an engine accessed a page for a documented automated purpose.
- Live user fetches show request-time access associated with a user action. They remain fetches, not clicks.
- Answer-level mentions and citations show what an engine selected and attributed in a returned answer.
- Recognized referrals show that a browser reached the measured site from an AI product.
- Conversions show whether the visit produced a signup, qualified lead, purchase, or another business outcome.
Keep crawler purpose and human outcomes separate
Foglift Sensor records crawler-purpose categories and recognized AI referral visits as separate event classes. AI Visibility adds answer-level mentions and provider-returned citations, so a team can diagnose where the path stopped instead of calling every fetch a win.
Limitations
- User-agent strings are self-declared and can be spoofed. This cohort classified requests by documented crawler user-agent tokens and did not independently verify provider network origin, so spoofed or misidentified requests may be present.
- The cohort contains 16 sites and 311 observed site-days. It is a descriptive product-data sample, not a representative internet panel.
- Installations began on different dates. Daily-rate correlations reduce exposure bias, but site size, content volume, audience, and crawl policy still differ.
- Referrer headers can be missing, suppressed, or altered. The report counts recognized AI referrers and does not claim complete attribution.
- A crawler category describes the documented role of the detected agent. It does not prove indexing, training inclusion, citation, recommendation, or a downstream conversion.
- Correlation is not causation. No intervention was randomized, and the study does not estimate the effect of allowing or blocking any crawler.
Definitions and reproducibility
- OpenAI, Overview of OpenAI Crawlers. Defines separate search, potential-training, and user-action agents.
- Anthropic, crawler controls for site owners. Defines Claude-SearchBot, ClaudeBot, and Claude-User.
- Perplexity, crawler documentation. Defines PerplexityBot and Perplexity-User.
- The reproducible aggregation is stored with the report as
state/research/ai-crawler-fetches-vs-referrals-2026-aggregation.sql. The public CSV contains aggregate rows only.
Frequently Asked Questions
Is an AI crawler fetch the same as an AI referral visit?
No. A crawler fetch is a server request from a recognized AI agent. A referral visit is a browser visit whose referrer identifies an AI product. The events have different causes, evidence, and business meaning.
Does a live user fetch prove that someone clicked through?
No. A live user fetch means an AI product requested a page during a user-associated action. It does not prove that the person opened the page in a browser, saw a citation, or converted.
How strongly did crawler traffic track referral traffic?
After normalizing each site by its observed days, total crawler fetch rate had a Pearson correlation of 0.04 with referral rate. Live user fetch rate was -0.09. Rank correlations were higher, at 0.43 and 0.36 respectively, but still did not make fetch counts a substitute for referral measurement.
Why were only 5.8% of crawler requests live user fetches?
Most recorded crawler traffic served other documented purposes. Training accounted for 49.9% of requests and indexing for 37.2%. The mix depends on each site's audience, crawl policy, content, and the agents active during the window.
What should an AI visibility dashboard measure separately?
At minimum, separate training, indexing, and live user fetches from answer-level brand mentions, provider-returned citations, recognized referrals, Google clicks, and conversions. Each layer answers a different question.
Can I download the aggregate study data?
Yes. The CSV contains the cohort totals, crawler-purpose mix, site outcome matrix, exposure-normalized correlations, and referral-engine totals. Customer domains, workspace IDs, paths, and site-level rows are excluded.
Free tool
Run a Technical Audit on your own site
Check technical SEO, performance, security, accessibility, and structural AI Readiness. No signup required.
Run Free Technical Audit