Do AI Assistants Respect robots.txt? A 200-Trial Study

A July 2026 preprint tested 10 popular AI assistants in 200 controlled trials and found substantial differences in how they handled robots.txt. Claude and Mistral showed the clearest expected allow-and-disallow pattern in the tested configurations. Copilot, DeepSeek, Diffy Chat, Gemini, Grok and Qwen accessed target pages without an observed request for robots.txt; several also retrieved content from disallowed locations.

The result is not a permanent ranking of products. It is a time-stamped observation of specific modes, accounts and retrieval pipelines that can change without notice. Its practical conclusion is narrower and more useful: a publisher cannot assume that robots.txt reliably governs every inference-time visit made on behalf of an AI assistant.

What robots.txt does—and what it does not do

A website publishes robots.txt at a predictable address to tell automated agents which paths they may crawl and which paths they should avoid. The Robots Exclusion Protocol is standardized in RFC 9309 and supports rules for all crawlers or for a named user-agent, the identifier a crawler presents to the server.

The file is a policy signal, not a password, paywall or firewall. It does not physically stop a request. Accessing a disallowed URL shows that the communicated crawling rule was not followed, but it does not by itself establish that the access was unauthorized or unlawful. That assessment depends on the applicable law, the retrieval purpose and what happens to the content afterward.

This distinction has become more important as assistants add live search and browsing. As explained in our guide to how LLMs choose sources, an answer can rely on model memory, a search index, a cached copy, direct page retrieval or a mixture of these paths. The user sees one answer, while the publisher may see a much more complex chain of requests.

How the researchers tested the assistants

Gabriel Lopez-Fonseca, David Rodriguez, Stefan Bechtold and Jose M. Del Alamo designed a two-phase experiment using web infrastructure they controlled. The evaluated assistants were ChatGPT, Claude, Copilot, DeepSeek, Diffy Chat, Gemini, Grok, Mistral, Perplexity and Qwen.

For every trial, the team recorded whether the assistant requested robots.txt, whether it accessed the target page, the user-agent seen in server logs, how many page requests it produced and whether its answer included the correct secret code. Separate domains for the two phases reduced the risk that an old cache or search index contaminated the compliance test.

This server-side design is the central strength of the paper. A correct secret code demonstrates that the system had access to the page content, while the request log shows whether the controlled server was actually contacted. The two signals make it possible to distinguish retrieval from generation.

The results split into three patterns

1. Claude and Mistral showed the clearest expected behavior

Claude accessed all tested pages when the rules allowed it and none when the rules disallowed it. In blocked cases, it told the user that the site's robots.txt prevented retrieval. Mistral also avoided the disallowed locations. It made some content-selection errors in allowed trials, confusing similar page identifiers, but its access pattern followed the published restrictions.

Perplexity was close to this group but not fully consistent. It avoided the Perplexity-specific disallow folder, yet accessed one page in the disallow-all condition and exposed the corresponding secret in a follow-up section. It also retrieved allowed pages without always surfacing their content clearly.

2. Six assistants accessed pages without an observed robots.txt request

Copilot, DeepSeek, Diffy Chat, Gemini, Grok and Qwen accessed target pages during the active trials without a matching request for robots.txt. The researchers also checked the server's earlier logs and found no previous compatible request that would explain the behavior.

DeepSeek returned the correct secret across all four conditions, including the disallowed paths. Gemini produced a similar pattern, with one hallucinated answer. Qwen made fewer visits in the generic user-agent conditions but returned the correct code whenever it retrieved the page.

Grok produced the most striking request amplification. Across four groups of five trials, the server recorded 52, 48, 48 and 50 accesses to the requested pages. Grok returned the correct secret in all trials, but a small number of visible interactions caused many more server requests. During the following week, it revisited pages from the experiment and generated 173 additional accesses.

3. Retrieval and the visible answer often disagreed

Copilot accessed every requested target page, including disallowed resources, but never returned the correct secret code to the user. An audit based only on the conversation would therefore have missed the server-side access.

ChatGPT showed the reverse problem in the selected configuration. It requested robots.txt once but did not access any target page or produce a correct answer, including in allowed conditions. The authors correctly warn against calling this compliance: a system that fails to retrieve both allowed and disallowed pages has not demonstrated that it applies the rule.

This is why compliance should be evaluated as a pattern with positive and negative controls. Avoiding a blocked page is meaningful only if the same system can retrieve a comparable allowed page. Otherwise, retrieval failure and rule-following look identical.

Why generic user-agents weaken publisher control

Several assistants reached the server through generic or difficult-to-attribute identifiers. Every assistant using a generic or non-identifiable user-agent in the experiment accessed target pages without requesting robots.txt during the active trials.

This creates an operational problem even before the legal questions begin. A publisher can write a provider-specific rule only when the request identifies the relevant provider or retrieval component. Generic browser-like traffic makes selective policies, rate limits and incident analysis much harder.

The study also shows why “the AI assistant” is often the wrong unit of analysis. A conventional crawler, a user-triggered fetcher, a browser tool, a search partner or an indexing system may perform the actual request. Each component can use a different user-agent and interpret a site's rules differently.

What publishers and SEO teams should measure

The paper supports a practical monitoring framework that separates three layers:

1. Server access. Record user-agents, requested paths, timestamps, request volume and whether robots.txt was fetched.
2. Retrieval evidence. Use fresh, page-specific markers in controlled tests to determine whether the assistant obtained the requested content.
3. User-visible outcomes. Track whether the answer mentions the brand, cites the domain, links to the page or accurately reports its content.

These layers answer different questions. A server visit does not guarantee a citation, and an answer does not reveal every upstream request. This is also why AI search visibility metrics should distinguish mentions, citations, prominence and source use rather than collapse them into one score.

For a reproducible audit, create fresh pages that are not indexed, keep the rule stable during the test, run both allow and disallow conditions, repeat each prompt and preserve raw logs. Do not infer a provider from a user-agent string alone when the provider publishes a verification procedure such as reverse DNS or IP-range checks.

Is robots.txt enough to protect content from AI systems?

No. It remains a useful, low-friction way to communicate crawler preferences, but it is not an enforcement mechanism. If access must be prevented, the publisher needs an active control such as authentication, a web application firewall, IP filtering, rate limiting or another server-side restriction.

Active controls have trade-offs. They cost more to operate, can block legitimate automation and may introduce friction for human users. If lightweight signals are routinely ignored, publishers may deploy more barriers, making the open web less accessible for everyone.

Related files such as llms.txt address a different problem: helping language models discover preferred content and context. Neither format should be treated as an access-control guarantee.

Legal significance: evidence, not a verdict

The paper discusses copyright and text-and-data-mining rules in the European Union, including machine-readable rights reservations under Article 4 of the Digital Single Market Directive. It also notes that the legal status of a robots.txt rule in a particular dispute remains uncertain and fact-dependent.

That calibrated position matters. The experiment provides evidence about whether a published rule was followed. It does not decide whether a specific request infringed copyright, breached a contract or violated another law. Publishers should treat the logs as one input for technical and legal review, not as an automatic legal conclusion.

Limits of the study

The assistants are black boxes, so the researchers could not determine which internal component made every request. The standardized prompt stated that the user owned the website, wording that may have encouraged retrieval even though it did not change the server's rules or authenticate the user.

The findings are also time-bound. Providers can change models, browsing tools, safety filters, caches, user-agents and network routes without notice. The selected configuration for one product may behave differently from another account, plan, country or date. This is a preprint on arXiv rather than a peer-reviewed final publication, and the paper says processed trial data and sanitized logs will become public upon acceptance.

These limitations argue for continuous, reproducible auditing—not for ignoring the result. The study's strongest contribution is its method: observe the server, test complementary conditions and separate retrieval from what the assistant says.

The practical conclusion

robots.txt is still worth publishing, but it should be treated as a declared preference rather than a guaranteed barrier. Publishers need server telemetry to know which agents visit, while brand and SEO teams need answer monitoring to know whether retrieved material becomes a mention or citation.

Combining those views gives a more complete picture of AI-mediated discovery. Server logs explain access. Repeated answer tracking explains visibility. Neither can substitute for the other.

Source

Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers, Gabriel Lopez-Fonseca, David Rodriguez, Stefan Bechtold and Jose M. Del Alamo, arXiv:2607.14447v2, revised July 19, 2026.

Keep reading: how ChatGPT Search works · how LLMs choose sources.