AI crawler analytics
Foglift shows which recognized AI agents requested your pages, when they arrived, and which paths they reached. Pair that access evidence with separate AI referral and prompt-citation reports so a crawler spike never gets mistaken for an answer win.
Observe the request layer
Most classic crawlers fetch HTML without executing your analytics JavaScript. A normal browser dashboard can therefore show zero crawler sessions while your server is handling GPTBot, ClaudeBot, OAI-SearchBot, or PerplexityBot requests. Foglift Sensor records recognized requests where they can actually be observed: in your application, middleware, edge worker, proxy, or server logs.
The report retains the named agent, requested path, and server-side time, then groups activity into useful windows. That makes it possible to see whether documentation, product pages, research, or stale URLs are being fetched. The report does not treat access as proof of visibility.
Access, referral, and citation data belong beside one another, but they should never be collapsed. Each signal has a different collection point and supports a different conclusion.
| Signal | Collection point | What it proves | What it does not prove |
|---|---|---|---|
| Crawler request | Server, edge, middleware, or access logs | A recognized agent requested a path at a recorded time. | That the page appeared in an AI answer or sent a visitor. |
| AI referral visit | Browser attribution from a recognized AI referrer | A person clicked from an identifiable AI surface to your site. | Which answer or source caused the click when the referrer omits it. |
| Prompt citation | A stored AI-engine answer and its cited URLs | A monitored answer cited the recorded page for that prompt and run. | That the cited page received a click or that every crawler fetch led to a citation. |
A crawler request is evidence of access. Even a user-triggered or citation-proximate fetch is not proof that an AI answer cited the page; verify citations from the stored answer and source URL.
A defensible workflow
Start at the layer that can observe the request, preserve the requested page, and compare like-for-like windows. Only then connect the activity to visibility or referral outcomes.
Step 1
Use a server, edge, middleware, WordPress, nginx, or supported framework integration so requests from non-JavaScript agents are observable. The browser pixel is useful for recognized referrals, but it cannot replace server-side crawler logs.
Step 2
Connect the Sensor to the intended hostname and test a known request path. Host binding keeps one site's traffic from being attributed to a different workspace and makes the page-level report trustworthy.
Step 3
Separate search, training, and user-triggered or citation-proximate agents. A GPTBot request and an OAI-SearchBot request come from the same provider but answer different operational questions.
Step 4
Review top paths, named agents, and request trends over consistent periods. Then compare those access signals with AI referrals and monitored citations before deciding which content or crawl policy to change.
A missing agent may reflect low discovery, a robots.txt decision, a CDN rule, or a server block. Compare policy with observed requests before changing access. A request that reaches a blocked response is still different from a successful page fetch.
Top-path data shows where agents already spend attention. Compare those URLs with current product facts, structured answers, and citation monitoring to find pages that are accessible but still fail to appear in answers.
Providers publish different agents for search, potential training, and user-triggered retrieval. Report the actual agent instead of turning every request from one provider into a single undifferentiated trend.
Foglift provides Sensor paths for Next.js, Node.js, Express, Cloudflare Workers, WordPress, nginx, and a browser pixel. Choose a server-side or edge path for crawler coverage, then use the browser layer for recognized AI referral visits. The workspace report keeps both available without treating them as the same event.
You can identify AI crawlers by observing HTTP requests at the server, edge, middleware, or proxy layer and matching verified user-agent patterns. Foglift groups recognized agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot while retaining the requested path and timestamp.
Install Foglift Sensor in the request path, connect it to the correct workspace hostname, and verify that a test request appears. A server-side or edge integration is required for complete non-JavaScript crawler coverage; a browser pixel alone cannot observe classic crawler requests that never execute page JavaScript.
The best fit depends on the evidence you need. Choose a tool that preserves named agents, requested paths, timestamps, status or access context, and fixed comparison windows. If you also need to connect access with outcomes, keep crawler requests, AI referral visits, and prompt citations as separate signals rather than merging them into one score.
No. A crawler request proves access to a path at a point in time. It is not proof that an AI answer cited the page. Citation evidence requires the answer and cited URL from a monitored prompt run, while referral evidence requires an identifiable visitor click from an AI surface.
Only when the agent executes JavaScript, which many crawlers do not. Use the pixel for recognized AI referral traffic and limited agent visits, and use a server, edge, middleware, or proxy integration for crawler-request coverage.
Install the Sensor, confirm the pages AI agents request, and compare that evidence with AI referrals and prompt-level citations in the same Foglift workspace.
Start free