When False Information Enters an AI System's Memory

Manipulating an AI answer does not always require attacking the model itself. If an assistant or fact-checking agent relies on web search or a retrieval-augmented generation system, an attacker can target the evidence layer: publish or inject material that is easy to retrieve, apparently credible and designed to support a false conclusion.

The July 2026 GPE preprint tests this risk under controlled conditions. Across 638 claims, every evaluated verification approach became less reliable as clean evidence was replaced with poisoned evidence. No method was consistently robust across all four attack types, and the most damaging attack preserved the style of real sources while changing decisive facts such as entities, amounts, dates and causal relationships.

A simple example: the wrong price repeated everywhere

Imagine a software company charges €49 per month. A collection of comparison pages, directories and articles begins to state that the price is €99. When an AI system retrieves several of those pages, the incorrect amount may appear to have support from multiple sources and enter the generated answer.

This is an illustrative scenario, not a documented incident from the GPE paper. The same mechanism could target more consequential claims: a company's financial health, a product-safety allegation, the identity of an executive, a fraud accusation or a medical quantity. GPE explicitly models attacks that alter people, organizations, dates, monetary values, causal relations and drug dosages while preserving source-like language.

Repetition makes the attack dangerous because many retrieval systems optimize for relevance, not truth. A highly relevant false passage can outrank a reliable but less optimized source. Several copies can also create the appearance of consensus unless the system detects that they share an origin.

What GEO poisoning means

Generative engine optimization, or GEO, aims to make content easier for generative search systems to retrieve, summarize and cite. In a benign strategy, that means publishing clear, well-sourced and machine-readable information. GEO poisoning applies similar visibility mechanics to misleading evidence.

The GPE authors define the threat as deliberately publishing large amounts of search-friendly and model-friendly content around a target claim or entity so that polluted evidence is more likely to enter the system's reasoning process. The verifier then faces a mixed evidence pool: some documents may be current and independent; others may be copied, outdated, altered or intentionally fabricated.

This is related to, but distinct from, a classic hallucination. A hallucination can arise without an external false document. GEO poisoning changes the information environment that the system uses as grounding. A fluent answer may therefore cite apparently relevant material and still be wrong.

Why RAG creates a new attack surface

Retrieval-augmented generation, or RAG, combines a model's parametric knowledge with external documents. A retriever finds passages related to a question, and the model uses those passages as context for its response. This makes answers fresher and easier to ground, but it transfers part of the system's reliability to the document collection and retrieval policy.

If a knowledge base, vector index or search result contains manipulated evidence, the system can retrieve the same error repeatedly. The poisoned page is no longer just a webpage someone might read; it becomes an input to an automated decision pipeline.

The risk increases when derived copies persist in search indexes, caches, vector stores, document repositories or other retrieval layers. Removing the original page does not guarantee that every representation disappears at the same time. This is one reason why teams should understand how LLMs choose sources and preserve provenance through ingestion and indexing.

What the GPE benchmark evaluated

The study used two backbone models, DeepSeek-V4-Flash and GPT-5.4, and compared four verification strategies: a direct model judgment, RAFTS, SAFE and STEEL. It tested four attack families:

• FakeGPT-style evidence: fluent, persuasive passages written to push an incorrect outcome.
• PoisonedRAG-style evidence: malicious documents injected into the retrieval corpus.
• Adaptive Tampering Attack (ATA): credible-looking source text in which crucial entities, amounts, dates or relations are changed.
• Ignore Injection: a document that poses as authoritative and instructs the verifier to ignore other evidence.

The results: clean accuracy did not predict robustness

Fact verification was already difficult without poisoning. The best exact six-way accuracy in the clean condition was 52.7% with DeepSeek-V4-Flash and 53.9% with GPT-5.4. Once contaminated evidence entered the pool, performance fell further and the most robust method changed with the attack.

Under full poisoning, no single verifier won across every condition. Resistance to instruction-like content did not imply resistance to altered facts. STEEL handled Ignore Injection comparatively well because its staged process could isolate instructions and gate uncertain evidence, yet it still degraded under FakeGPT, PoisonedRAG and ATA.

ATA was the most damaging attack on average. Across the two backbones and all methods, full-poisoning accuracy averaged 9.2% under ATA, compared with 11.6% for FakeGPT, 12.2% for PoisonedRAG and 14.6% for Ignore Injection. The authors attribute ATA's effectiveness to a difficult pattern: it changes the decisive fact while preserving plausible context and source-like style.

The benchmark also found a dose-response relationship. As one, two and then all three clean evidence objects were replaced, accuracy generally declined. More reasoning steps did not automatically solve the problem; a method can decompose a claim and still judge each part from contaminated evidence.

“AI memory” contains several different layers

The word memory is useful, but it can hide important technical differences. A fact may exist in at least three places:

1. Temporary context: passages retrieved for one answer and discarded after the interaction.
2. Retrieval infrastructure: a cached copy, search index, vector database or document store that may be reused for later answers.
3. Model parameters: patterns learned during training or subsequent model improvement.

GPE evaluates the first layer: the evidence supplied during fact verification. It also provides a graph for studying source dependence, evidence reuse and possible propagation paths. It does not demonstrate that poisoned web pages automatically enter model training or become encoded in model weights.

Training-data poisoning is a separate lifecycle threat recognized in adversarial machine-learning taxonomies. If unwanted information has influenced model parameters, deleting the source document may not remove that influence. Correction may require filtering, fine-tuning, retraining or machine-unlearning techniques. Research such as Google's MUSE benchmark shows that reliable language-model unlearning remains difficult: methods can damage general utility, leak information or fail under repeated and large-scale removal requests.

Defensive GEO: monitor the evidence, not only the ranking

GEO is often framed as a visibility discipline: get a brand mentioned in ChatGPT, Gemini, Claude or Perplexity. The GPE findings show why it also needs a defensive layer. A company should monitor not just whether it appears, but whether the system's underlying description remains accurate.

A practical defensive workflow includes:

1. Define critical facts. Maintain a dated record of prices, product names, executive roles, safety claims, locations and policies.
2. Run repeated prompts. Test the same material questions across engines and dates; one answer is only one sample.
3. Capture cited sources. Record the URLs and domains behind each material claim, not just the final wording.
4. Check independence. Look for syndication, copied text, common ownership and several pages that trace back to one origin.
5. Compare decisive facts. Extract amounts, names, dates, dosages and causal statements instead of relying on overall semantic similarity.
6. Escalate persistence. If an error remains after the original page is corrected, investigate cached, indexed and third-party copies.

This complements standard AI search visibility metrics. Mentions and share of voice measure exposure; factual accuracy, source provenance and contradiction rates measure information integrity.

What this study does not prove

GPE is a benchmark, not a measurement of how often real brands are attacked on the open web. Its poisoning ratios deliberately control an evidence set of three documents; a 100% condition means all three were replaced in that experiment, not that the public internet is fully contaminated.

The paper is also an arXiv preprint and its implementation had not yet been released when version 1 was posted. The evaluated models, prompts, evidence selection and verification methods define the scope of the result. Other systems may behave differently.

Most importantly, the study tests retrieval-time evidence, not long-term model training. Claims about an error entering the model's parameters require separate telemetry and experiments. Keeping that boundary clear prevents a real security finding from turning into a broader claim the evidence does not support.

A new layer of digital reputation

Companies have long monitored Google results, reviews and social mentions. They now need to monitor the version of the brand assembled by AI systems: which facts appear, which sources support them and whether apparently independent pages are really repeating one narrative.

A false claim can be concise, confident and well cited. That presentation does not make the evidence independent or true. Defensive GEO means protecting a company's information environment from false data, manipulated content and coordinated repetition, then measuring whether corrections actually reach the generated answer.

Sources

• GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning, Zhaoqi Wang et al., arXiv:2607.20730v1, July 22, 2026.
• PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models, USENIX Security 2025.
• Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025.
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models, ICLR 2025.

Keep reading: do AI assistants respect robots.txt? · what to do when AI invents facts about your brand.