Why One AI Answer Is Not a Visibility Measurement

One generated answer is not a reliable measurement of brand visibility. In repeated Bee LLM observations, textual brand presence changed across runs in 60.6% of eligible ChatGPT prompt cells, 50.0% of Gemini cells and 71.1% of Perplexity cells. A brand could appear for a prompt in one response and be absent in another under the observed sample.

The finding does not mean every answer is random, and presence does not mean recommendation. It shows that a single yes-or-no check can misclassify a variable prompt. Visibility measurement needs repeated observations, explicit denominators and a distinction between an observed absence and evidence that the brand never appears.

What changed across repeated answers

A “mixed” prompt cell contained at least one response with a textual own-brand match and at least one without it. ChatGPT had one prompt where the brand was always present, 40 mixed prompts and 25 where it was never present. Gemini had five always, 33 mixed and 28 never. Perplexity had two always, 32 mixed and 11 never.

These states describe only the eligible cohort and observed window. A “never” cell means no presence was detected in the collected valid responses, not that the engine could never mention that brand in the future or under another context.

How presence was defined

Presence meant a literal, whole-token match for a normalized alias of the project's own brand in response text. Aliases shorter than four characters were discarded to reduce ambiguous matches. All 11 projects had at least one usable alias, so no otherwise eligible response was excluded for missing alias coverage.

The definition is intentionally narrow and auditable. It does not infer a brand from a product description, logo, URL or pronoun. It also does not decide whether the mention was positive, negative, prominent or a recommendation. A sentence criticising the brand and a sentence selecting it as the first option both count as textual presence.

As a quality check, textual matching agreed with the structured mention table in 99.1% of ChatGPT records, 95.6% of Gemini records and 98.5% of Perplexity records. Agreement supports the operational detector, but it does not create a human-coded gold standard or change the scope of the outcome.

The frozen cohort and unit of analysis

The snapshot was frozen at 18:00 UTC on 21 July 2026. Demo accounts, incomplete executions, empty responses and responses with errors were excluded. A prompt-engine cell required at least five valid answers. The final engine cuts exceeded the publication gates of ten executions, five prompts, three projects and three accounts.

The atomic unit was a prompt-engine-execution observation with binary textual presence. The prompt was the primary cluster. Rates and Wilson 95% intervals were calculated per prompt rather than treating all response rows as independent market observations.

The cohort mixes the prompts used by eligible Bee LLM projects, including different query types. It is not a random sample of consumer demand. The study therefore reports variability inside the cohort and does not translate a mean presence rate into market share.

The risk of a one-run false absence

To illustrate the sampling risk, the analysis considered only prompt cells in which the brand appeared at least once. For each cell, it used the observed presence rate p. Under the descriptive plug-in calculation, the chance of seeing no presence in k checks is (1-p)^k. The cell results were then summarized without claiming a forecast.

With one check, the mean all-absent probability was 68.1% for eligible ever-present ChatGPT cells, 51.7% for Gemini and 49.0% for Perplexity. With five checks it fell to 34.1%, 22.3% and 13.2%, respectively. With ten, the descriptive values were 20.0%, 12.1% and 5.1%.

The simulation used 100,000 draws with fixed seeds as a check and closely reproduced the plug-in values. It does not turn the observed rates into stable engine probabilities. Repeated answers can be dependent, behaviour can change over time, and the empirical rate itself is uncertain.

A concrete interpretation

Suppose a brand appears in three of ten valid answers for one prompt. One analyst checks a single response and happens to see an absence. That observation is correct for the run, but “the brand is not visible for this prompt” is too broad. The sample supports “the brand was absent in one observed answer.”

If the analyst collects five answers and sees presence twice, the result becomes a sampled rate of two out of five with a wide uncertainty interval. It still does not estimate every user experience, but it is more informative than a binary label. Repeating the same inventory later can show whether the distribution changed or whether the first run was an outlier.

The discipline is linguistic as well as statistical: say “observed presence,” “sample mention rate” and “valid responses.” Avoid “the engine ranks us,” “we are invisible” or “we own 30% visibility” unless the underlying unit and denominator actually justify those claims.

How many repetitions are enough?

There is no universal minimum that makes every prompt stable. The right design depends on the decision, variability, cost and acceptable uncertainty. A directional weekly monitor may use fewer repetitions across more prompts; an investigation into an important high-intent prompt may use more repetitions and qualitative review.

Start by setting the precision target. If the team needs to distinguish a 10% presence rate from 50%, a handful of runs can be useful. If it needs to detect a small movement, the sample must be larger and repeated over comparable periods. Report confidence intervals and collection failures rather than presenting the point estimate alone.

Allocation matters too. Ten repetitions of one prompt cannot compensate for ignoring the rest of the buyer journey. Balance problem, category, comparison and selection prompts, then repeat within each segment. The workflow for tracking brand mentions in AI search explains how to manage aliases and prompt inventories.

A defensible visibility sampling workflow

1. Define the decision. Specify whether the programme tracks discovery, comparison presence, representation problems or source exposure.
2. Freeze prompts and conditions. Store exact text, engine, locale, location, mode and observation time.
3. Repeat observations. Use a documented cadence and avoid selecting only runs that show the brand.
4. Separate failures from absence. An errored or empty response belongs in a coverage metric, not in the visibility denominator.
5. Match aliases conservatively. Test common names and short aliases for false positives; retain a review trail.
6. Aggregate within prompt. Calculate rates and intervals before combining segments or engines.
7. Read answer examples. Presence alone cannot tell whether the representation is helpful, accurate or negative.

The broader AI visibility measurement framework places sampled presence beside prominence, citations, sentiment and traffic. A composite score should never erase the response count or make missing observations look like absence.

What the study does not prove

The mixed rates do not establish the cause of variation. Differences could accompany model sampling, search results, context, provider changes or other unobserved conditions. The study does not manipulate those factors and cannot assign causality.

Presence is not recommendation quality. A literal alias match can occur in a disclaimer, comparison, negative reference or factual statement. Teams should review answer context before deciding whether more presence is desirable or whether corrective content is needed.

The plug-in false-absence exercise is conditional on cells where presence occurred at least once. It deliberately excludes always-absent cells and should not be presented as the chance that any random brand will be missed. It uses empirical rates from the same data and serves as an illustration of sampling risk, not an external prediction.

Finally, the eligible Bee LLM projects and prompts are not a probability sample of the market. The numbers describe the frozen cohort through the cutoff. Different industries, languages, brands, product surfaces and future model versions may produce different variability.

Evidence and reproducibility

The production extraction ran in a repeatable-read, read-only transaction. Aliases, brands and response text stayed in remote memory and were not emitted. The methodology, analysis wrapper, CSV and aggregate are checksummed. The aggregate used here has SHA-256 d8fde95893ef3830920e6c114cf827a8d9306223bfce9ce34fce2a3cb28b0a87.

The conclusion is operational: a single answer can document what happened once, but it cannot by itself measure a distribution. In this cohort, half or more of eligible prompt cells in each studied engine showed mixed brand presence. Repeated, transparent sampling turns that volatility from a hidden source of error into something a team can quantify and manage.