What Changes When ChatGPT Searches? Paired Brand Recommendations Across 15 Industries
A paired test across broad category and specific capability questions shows when web retrieval changes the brands ChatGPT recommends.
Methodology
Foglift tested 30 brand-neutral buyer questions across 15 industries twice: one separate answer without web search and one with web search. Search-on calls captured query text, returned source URLs, and citations. Products and plans were normalized to one commercial entity; requested integrations, medications, publishers and retailers were excluded. The paired one-shot design measures observed overlap and source-pool co-occurrence, not selection order or causality.
What changed when search was available?
Search changed the mix most when the buyer asked for a specific capability. Broad category answers stayed closer to the brands ChatGPT already knew. Specific questions gave live web retrieval more room to change the shortlist.
What search on and search off mean
Search off: ChatGPT answers from what it memorized in training. Nothing on the web today changes that list until a new model ships.
Search on: ChatGPT runs one or more web queries first, reads the pages it gets back, then answers. In the API this is the web_search tool. In the consumer app the model decides on its own whether to search.
A brand that is missing from the memorized list has one way into today's answer: it must be read during a search. Broad questions are mostly answered from memory. Specific questions are where search changes the answer. That is why the study pairs the two modes on every question. Memory is the long game. Retrieval is the game you can win this quarter.
Across 30 buyer questions, 50 of 71 distinct commercial entities in search-on answers also appeared in the matching no-search answer. The remaining 21 appeared only in the search-on answer. Broad questions retained 81.2% of their search-on entities from the paired search-off answer; specific capability questions retained 47.8%.
50 of 71 search-on entities appeared in both paired answers
39 of 48 entities appeared in both paired answers
11 of 23 entities appeared in both paired answers
For entities that appeared only in the search-on answer
Broad and specific questions split sharply
Broad questions asked for a category shortlist, such as a CRM for a small B2B startup. Specific questions added a capability or constraint, such as native WhatsApp integration and two-way contact sync.
Each row compares the brands ChatGPT named with search on against the brands it named with search off for the same questions. Overlapping brands appeared in both.
| Question group | Paired answer comparison | Search-on retrieval evidence | ||||
|---|---|---|---|---|---|---|
| Brands named with search on | Brands named with search off | Overlapping brands | Brands search on only | Brands search off only | Search-on-only brands whose own page was read | |
| All 30 questions | 71 | 88 | 50 · 70% | 21 | 38 | 19 of 21 |
| Broad category (15) | 48 | 62 | 39 · 81% | 9 | 23 | 7 of 9 |
| Specific capability (15) | 23 | 26 | 11 · 48% | 12 | 15 | 12 of 12 |
With search on, ChatGPT dropped 38 of the 88 brands it had named from memory and added 21 new ones. Searching made the shortlist shorter and more specific, rather than simply making it longer. These pooled values describe the frozen sample, not a stable rate for any one prompt.
Official pages co-occurred with new recommendations
Nineteen of the 21 search-on-only entities had a URL from their reviewed official domain in the returned source pool. The split was 12 of 12 for specific questions and 7 of 9 for broad questions.
This is useful without turning it into a causal claim. It shows that a clean manufacturer page can be present when a new option enters a search-on answer. It does not show whether the page caused selection, whether the entity was already under consideration, or what would happen in a second run.
Specific capability work
Publish one authoritative page that states the capability, price, plan boundary, compatibility, and operating limit in complete sentences. Specific prompts produced the larger search-on-only share in this study.
Broad category work
Earn independent reviews, roundups, community references, and other third-party corroboration. Broad consumer-product questions relied much more heavily on publisher and review citations.
Search queries often named an existing option
23 of 30 search-on runs issued at least one query containing a commercial entity from the paired no-search answer. Twenty-five runs issued at least one site: query, with 130 such queries in total.
Those query traces narrow the interpretation further. Retrieval can verify an option already present in the paired no-search answer, find a different option, or do both. Query text alone does not establish the point at which a recommendation was chosen.
Results by sector and question type
Broad Tech SaaS questions had the highest share of overlapping brands. Specific consumer-service questions moved furthest toward search-on-only entities. Broad CPG and retail questions had the clearest third-party source pattern.
Search off: ChatGPT answers from memory only. Search on: it can read the web before it answers. Four out of five brands in the search-on answer were the ones it had already named with search off. Tech SaaS is the hardest wall: 16 of 17.
Each row compares the brands ChatGPT named with search on against the brands it named with search off for the same questions. Overlapping brands appeared in both.
| Sector and question type | Paired answer comparison | Search-on retrieval evidence | |||||
|---|---|---|---|---|---|---|---|
| Brands named with search on | Brands named with search off | Overlapping brands | Brands search on only | Brands search off only | Search-on-only brands whose own page was read | Citations to brand sites / to third parties | |
| Tech SaaS, broad | 17 | 23 | 16 · 94% | 1 | 7 | 1 of 1 | 20 / 0 |
| Tech SaaS, specific | 7 | 10 | 4 · 57% | 3 | 6 | 3 of 3 | 10 / 0 |
| Consumer services, broad | 16 | 21 | 11 · 69% | 5 | 10 | 5 of 5 | 13 / 3 |
| Consumer services, specific | 8 | 8 | 3 · 38% | 5 | 5 | 5 of 5 | 9 / 2 |
| CPG and retail, broad | 15 | 18 | 12 · 80% | 3 | 6 | 1 of 3 | 5 / 15 |
| CPG and retail, specific | 8 | 8 | 4 · 50% | 4 | 4 | 4 of 4 | 14 / 0 |
Most citations pointed to a recommended entity's site
Seventy-one of 91 citations, or 78.0%, matched the official domain of a reviewed recommended entity in the same answer. Specific questions produced 33 first-party and two third-party citations.
Broad CPG and retail questions were the exception. Five of their 20 citations went to a recommended entity's site, while 15 went to publishers and reviewers. A manufacturer source of truth and independent corroboration solve different retrieval jobs.
What this changes for AI visibility work
- Separate broad and specific prompts. A category shortlist and a capability-constrained buyer question should not share one visibility target or one success rate.
- Give specific questions an exact manufacturer answer.State capabilities, limits, pricing, integrations, and eligibility on the page an answer engine should retrieve.
- Build independent category association. Broad questions can lean on reviewers and publishers even when product pages are strong.
- Measure outcomes separately. A source-pool appearance, crawler fetch, citation, brand mention, referral click, and Google Search click are different events.
Foglift supports that separation directly. The permanent free Technical Audit evaluates page readiness without signup. Launch starts at $49 per month and adds monitoring across five engines, three brands, and developer access through the API, CLI, and MCP. See the monitoring workflow, developer surface, and crawler and referral measurement.
Limits
Each prompt ran once per mode, so individual answers can vary. The 15 specific prompts were written for this study and do not represent every buyer constraint. Source-pool presence does not reveal selection order or causality. The API setup is also distinct from the consumer ChatGPT application.
The defensible conclusion is observational: broad answers in this sample repeated more of the paired no-search entity set, while specific answers added more search-on-only entities, and most of those added entities had an official-domain page in the returned source pool.
Frequently Asked Questions
Does web search completely change ChatGPT brand recommendations?
Not in this frozen sample. Fifty of 71 distinct commercial entities in search-on answers also appeared in the paired no-search answer, an overlap of 70.4%. The result describes 30 one-shot answer pairs and is not a universal rate.
Did broad and specific buyer questions behave differently?
Yes. Broad category questions had 81.2% paired entity overlap, while specific capability questions had 47.8% overlap. Specific questions produced 12 search-on-only entities, compared with nine for broad questions.
Did official product pages appear in the retrieval pool?
Yes. Nineteen of 21 search-on-only entities had a page on their reviewed official domain in the returned source pool. That included all 12 search-on-only entities from specific questions and seven of nine from broad questions.
Does an official page in the source pool prove it caused the recommendation?
No. Source-pool presence is co-occurrence. The paired API output does not reveal whether an entity was selected before or after retrieval, whether a page caused its inclusion, or whether a different page would change the answer.
What should a brand do with this finding?
Treat broad category association and specific capability retrieval as separate jobs. Publish exact manufacturer pages with prices, limits, and operating details for specific questions, then earn independent category mentions for broad discovery questions. Measure answers, citations, referrals, and search clicks separately.
Can I download the study data?
Yes. The downloadable CSV contains all 30 prompts, normalized entity sets, counts of overlapping brands named in both modes, search-on-only counts, source-pool matches, query flags, citation types, and retrieval-pool sizes.
Free tool
Run a Technical Audit on your own site
Check technical SEO, performance, security, accessibility, and structural AI Readiness. No signup required.
Run Free Technical Audit