Search
OAI-SearchBot, Claude-SearchBot, PerplexityBot
The provider requested content for search discovery or search-result quality.
Access does not prove that the page was indexed, selected, or cited.
AI crawler analytics
Track which AI crawlers are visiting your website: GPTBot, ClaudeBot, and OAI-SearchBot
Foglift shows which recognized AI agents requested your pages, when they arrived, and which paths they reached. Pair that access evidence with separate AI referral and prompt-citation reports so a crawler spike never gets mistaken for an answer win.
GPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-SearchBotPerplexityBot
Purpose methodology
Search, Training, and Live answer fetch are different events
Foglift classifies a recognized request by the agent's documented job. That keeps a provider's search crawler, potential-training crawler, and user-triggered fetcher from being reported as one interchangeable traffic number.
Training
GPTBot, ClaudeBot
The provider requested public content that may contribute to model development.
A request does not prove that the content entered a training set or changed a model.
Live answer fetch
ChatGPT-User, Claude-User, Perplexity-User
A page was pulled during a user-triggered AI answer or action.
The fetch is citation-proximate, but it is not proof that the answer cited the page.
Multi-purpose and unclassified agents stay separate instead of being forced into one of these three categories.
Provider definitions checked September 7, 2026
The classifications above follow the current crawler documentation from OpenAI, Anthropic, and Perplexity.
Observe the request layer
See access that browser analytics misses
Most classic crawlers fetch HTML without executing your analytics JavaScript. A normal browser dashboard can therefore show zero crawler sessions while your server is handling GPTBot, ClaudeBot, OAI-SearchBot, or PerplexityBot requests. Foglift Sensor records recognized requests where they can actually be observed: in your application, middleware, edge worker, proxy, or server logs.
The report retains the named agent, requested path, and server-side time, then groups activity into useful windows. That makes it possible to see whether documentation, product pages, research, or stale URLs are being fetched. The report does not treat access as proof of visibility.
Questions the report can answer
- Who arrived? Compare recognized agents and their documented roles.
- What did they request? Rank the paths receiving crawler attention.
- When did activity change? Compare consistent 24-hour, 7-day, 28-day, or 3-month windows.
- Did access become an outcome? Check referral and citation evidence separately.
Three signals, three different claims
Access, referral, and citation data belong beside one another, but they should never be collapsed. Each signal has a different collection point and supports a different conclusion.
| Signal | Collection point | What it proves | What it does not prove |
|---|---|---|---|
| Crawler request | Server, edge, middleware, or access logs | A recognized agent requested a path at a recorded time. | That the page appeared in an AI answer or sent a visitor. |
| AI referral visit | Browser attribution from a recognized AI referrer | A person clicked from an identifiable AI surface to your site. | Which answer or source caused the click when the referrer omits it. |
| Prompt citation | A stored AI-engine answer and its cited URLs | A monitored answer cited the recorded page for that prompt and run. | That the cited page received a click or that every crawler fetch led to a citation. |
A crawler request is evidence of access. Even a user-triggered or citation-proximate fetch is not proof that an AI answer cited the page; verify citations from the stored answer and source URL.
A defensible workflow
Turn raw agent requests into useful evidence
Start at the layer that can observe the request, preserve the requested page, and compare like-for-like windows. Only then connect the activity to visibility or referral outcomes.
Step 1
Install at the request layer
Use a server, edge, middleware, WordPress, nginx, or supported framework integration so requests from non-JavaScript agents are observable. The browser pixel is useful for recognized referrals, but it cannot replace server-side crawler logs.
Step 2
Verify the workspace host
Connect the Sensor to the intended hostname and test a known request path. Host binding keeps one site's traffic from being attributed to a different workspace and makes the page-level report trustworthy.
Step 3
Segment agents by purpose
Separate search, training, and user-triggered or citation-proximate agents. A GPTBot request and an OAI-SearchBot request come from the same provider but answer different operational questions.
Step 4
Compare stable windows
Review top paths, named agents, and request trends over consistent periods. Then compare those access signals with AI referrals and monitored citations before deciding which content or crawl policy to change.
Audit crawl policy
A missing agent may reflect low discovery, a robots.txt decision, a CDN rule, or a server block. Compare policy with observed requests before changing access. A request that reaches a blocked response is still different from a successful page fetch.
Prioritize crawled pages
Top-path data shows where agents already spend attention. Compare those URLs with current product facts, structured answers, and citation monitoring to find pages that are accessible but still fail to appear in answers.
Separate provider roles
Providers publish different agents for search, potential training, and user-triggered retrieval. Report the actual agent instead of turning every request from one provider into a single undifferentiated trend.
Sensor install matrix
Install at the deepest layer your stack allows
Server, edge, middleware, and proxy paths observe non-JavaScript crawler requests. Browser paths observe recognized referral visits and only the agents that execute JavaScript. Pick the row that matches your stack and the evidence you need.
| Stack | Collection layer | Evidence available |
|---|---|---|
| Next.js | Middleware or server | Crawler requests and recognized AI referrals |
| Cloudflare Workers | Edge worker | Crawler requests and recognized AI referrals |
| nginx | Reverse proxy | Crawler requests before the application layer |
| WordPress | Plugin plus browser pixel | Requests that reach WordPress and recognized AI referrals |
| Browser pixel | Browser | Recognized AI referrals and agents that execute JavaScript |
| Shopify | Browser pixel | Recognized AI referrals |
| Webflow | Browser pixel | Recognized AI referrals |
| Framer | Browser pixel | Recognized AI referrals |
| Wix | Supported app path | Recognized AI traffic through the supported connection |
Plan boundaries
The Sensor starts on Free
Install the Sensor and inspect crawler activity without moving to a paid plan. Upgrade when you need broader answer monitoring, faster cadence, or developer access.
Free
$0
Sensor, AI Crawler Analytics, unlimited single-page Technical Audits, and active-use weekly Perplexity monitoring
Launch
$49/mo
Free capabilities plus daily five-engine monitoring, full-site Technical Audits, REST API, CLI, MCP, and webhooks
Growth
$129/mo
Launch capabilities plus twice-daily monitoring and Buyer-intent Win Rate
Enterprise
Custom
Growth capabilities plus hourly monitoring, custom allowances, and full white-label delivery
See the full allowances and annual billing options on the pricing page.
Frequently asked questions
Which AI crawlers are visiting my website?
You can identify AI crawlers by observing HTTP requests at the server, edge, middleware, or proxy layer and matching verified user-agent patterns. Foglift groups recognized agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot while retaining the requested path and timestamp.
How do I track GPTBot, ClaudeBot, and OAI-SearchBot traffic?
Install Foglift Sensor in the request path, connect it to the correct workspace hostname, and verify that a test request appears. A server-side or edge integration is required for complete non-JavaScript crawler coverage; a browser pixel alone cannot observe classic crawler requests that never execute page JavaScript.
What is the best AI crawler analytics tool?
Foglift is a strong fit for developer-led teams that need crawler requests, AI referrals, Technical Audits, and answer monitoring in one workspace. The Sensor recognizes 28 named crawler and agent patterns. AI Crawler Analytics reports visits by agent and path across 24-hour, 7-day, 28-day, and 3-month windows while keeping access separate from citation proof. The Free plan includes the Sensor and unlimited single-page Technical Audits. Launch costs $49 per month and adds daily monitoring across ChatGPT, Claude, Perplexity, Gemini, and Google AI Overview, plus API, CLI, and MCP access.
Does an AI crawler visit mean my page was cited?
No. A crawler request proves access to a path at a point in time. It is not proof that an AI answer cited the page. Citation evidence requires the answer and cited URL from a monitored prompt run, while referral evidence requires an identifiable visitor click from an AI surface.
Can a browser analytics pixel detect AI crawlers?
Only when the agent executes JavaScript, which many crawlers do not. Use the pixel for recognized AI referral traffic and limited agent visits, and use a server, edge, middleware, or proxy integration for crawler-request coverage.
Connect crawler access to answer outcomes
Install the Sensor, confirm the pages AI agents request, and compare that evidence with AI referrals and prompt-level citations in the same Foglift workspace.
Start free