AI & Machine Learning

200 OK Is Not a Read: One in Five Successful Fetches Contains No Page Content

Leo Wang November 16, 2026
200 OK Is Not a Read: One in Five Successful Fetches Contains No Page Content

200 OK Is Not a Read: One in Five Successful Fetches Contains No Page Content

Every liveness check in this industry runs on the same shortcut. Fetch the URL, look at the status line, and treat 200 as success. Dead-link audits do it. Citation-drift studies do it. Agent fetch tools do it. Your own "can AI read this page" spot-check almost certainly does it.

A measurement published on 31 October 2026 put a number on how often that shortcut is wrong. Across 10,307 fetches over three days, 20.9% of the pages that a static client receives with status 200 do not contain the page's main content [1]. Not blocked, not missing — answered, with a body, and the body is an application shell, a challenge widget, a block notice, a homepage served in place of a deep link, a parked placeholder, or a consent wall.

Google says the same thing about its own pipeline in one sentence: an HTTP 2xx status code does not guarantee indexing, and where the content turns out to be an empty page or an error message, Search Console reports it as a soft 404 [2]. The failure class is not exotic. It just has no status code of its own, so nothing that reads the status line can see it.

What follows is what the measurement found, why your CDN's own blocking is a different problem, and the cheapest way to detect the difference.

---

Quick answer

  1. A 200 can contain seven different things, and six of them are not your page. The taxonomy separates content from block pages, challenges, shells, error or substitute pages, parked pages and consent walls [1].
  2. The rate varies by a factor of nearly three across populations. Unusable share of 200 responses runs from 9.8% for links listed in llms.txt files to 26.3% for the Tranco top 10k, with cited sources at 23.2% [1].
  3. This is a separate failure from the one your CDN produces. Cloudflare's edge block and challenge pages return 403, and its rate-limiting block returns 429 [3]. Those are visible to any status-based check. The 200s are not.
  4. The error compounds downstream. On 6,156 claim–source pairs, reading every 200 as content inflated the apparent silent citation-drift rate from 5.8% to 21.9% — 3.8 times as much apparent drift [1].
  5. Cheap checks catch most of it. Redirect-to-root, probe comparison, or fewer than 930 characters of visible text — combined and tuned on held-out sites — recovered 86% of unusable responses at a false-alarm rate near 5% [1].

The boundary on all of it: this is arXiv:2611.00884v1, a single-author preprint with no peer review and no venue, collected from one vantage point over three days in October 2026. The types it found will recur. The exact percentages belong to that setup.

---

What a 200 can actually contain

The paper's unit of analysis is one fetch of one URL by an automated client. A response other than 200 gets an explicit outcome: dead (404, 410, no DNS record), blocked (401, 403, 429), or failed. A 200 gets one of seven types, and the counts below cover all 4,560 static 200 responses in the study, pages and random-path probes together.

TypeWhat the body containsCount
A contentthe main content of the requested page3,575
D shellnavigation, skeleton or application frame without the main content354
E error or substitutean error page, or a different page such as the homepage341
C challengea JavaScript check, bot check or captcha159
B blockan access-denied page83
F parkeda parked domain or hosting placeholder27
G consent wallonly a cookie or consent gate21

Counts from Table 1 of the preprint [1].

Two design decisions in that taxonomy matter for how you read the rest.

The types are defined by what a reader of the response can use, not by the server's intent. A geographic block and a bot block are both type B. A login page served in place of a deep link is type E. The question is never "did the server mean to serve this" but "did the agent end up with the page".

And type A can still be incomplete. To catch that, every static fetch was paired with a rendered fetch of the same URL, and the paper computes retention — the share of the rendered page's sentences and numeric facts that appear in the static text, averaged over the unit kinds present, after removing consent-banner text. A static 200 counts as unusable if its type is B through G, or if both responses are type A and retention falls below 0.5, or if it answers a random non-existent path.

That last clause is a soft 404 by construction, and it is worth sitting with: the study generated random 16-hexadecimal-character paths on every site specifically to find out which servers answer URLs that cannot exist.

The 0.5 cut-off is not doing much work. At retention below 0.3 the unusable share of static 200 pages is 20.0%; at below 0.7 it is 23.5% [1].

How often it happens, and to whom

Of all 7,293 pages fetched, 60.2% returned 200 to the static client. The table below splits what happened inside those 200s.

PopulationPagesReturned 200Unusable share of those 200s95% CISoft-404 probe answered 200
Links listed in llms.txt files59590.6%9.8%7–1310.5%
Tranco top 10k63169.4%26.3%22–3113.3%
Markdown-serving sites in the top 10k18691.4%17.1%11–2417.3%
Tranco 10,001–100,00062266.1%20.9%17–2513.7%
Tranco 100,001–1,000,00046471.1%16.4%12–2113.1%
URLs cited in ExpertQA4,79552.1%23.2%19–28—

All figures from Table 3 of the preprint; confidence intervals resample registrable domains 2,000 times [1].

Read the first row carefully before drawing any comfort from it. The llms.txt population looks far healthier than the rest, and the paper says why: that stratum was drawn only from sites whose llms.txt listed at least two links that a public corpus had already recorded as answering 200, so it is biased toward working links [1]. It is a floor, not a verdict on the format.

The worst three populations are the Tranco top 10k, the 10k–100k band, and the cited sources. The most popular sites on the web are the ones most likely to answer with something other than their page.

What the failures are differs by population in a way that changes what you should check. Shells and partial content are the largest group in every Tranco band — an application frame with navigation and footer and no article. Among the cited sources, challenge and block pages are 8.2% of 200 responses, and four document and scholarly hosts account for 67% of them: scribd.com, mdpi.com, biomedcentral.com and springer.com [1]. If your pipeline cites publishers, your rate is a property of which publishers.

Two sanity checks on the labels, because the headline number comes from an LLM labeler. The automatic labels were checked against 300 items reviewed blind by a different model family, and that review lowered the test-split estimate from 25.5% to 20.0% (95% CI 13.3–27.7%). An author then labeled 60 of the reviewed pages blind, agreeing with the review on 47 of 57 decidable items — 82.5% raw, Cohen's κ of 0.64 on usability. Agreement on the specific type was much weaker, κ of 0.26 [1]. So treat "usable or not" as reasonably solid and "which of the seven types" as indicative.

Your CDN's blocking is a different problem

There is a tempting misreading here: that this measurement is about bot blocking, and therefore an argument for loosening your edge rules. It is not, and the status codes prove it.

Cloudflare documents its own error page types explicitly. WAF block returns 403. IP or country block returns 403. IP or country challenge returns 403. Managed challenge and I'm Under Attack Mode return 403. Rate limiting block returns 429 [3].

Every one of those is visible to a status-based check. If you turn on an aggressive edge rule, a crawler sees a 4xx and records a failure. That is the honest case.

So where do the 159 challenge pages that arrived with status 200 come from? The paper's examples name Cloudflare Turnstile alongside a publisher "Client Challenge" and an image captcha. Turnstile is a different mechanism from an edge challenge page, and Cloudflare says so: Turnstile can be embedded into any website without sending traffic through Cloudflare, and in contrast to the challenge page offerings it lets an operator run challenges anywhere on the site without requiring the use of Cloudflare's CDN [4]. A page gated by an embedded widget is the publisher's own 200 response that happens to contain a gate instead of an article.

The same logic covers the rest of the taxonomy. A shell is your framework's normal output. A soft 404 is your CMS answering politely. A consent wall is your privacy vendor. None of these are your security configuration, and none of them emit a status code that says anything is wrong.

This is also precisely the gap the adjacent literature leaves open. A measurement published four days later quantified denial rates across 2,000 sites and found standard Chromium automation refused on 0.2% of homepages but 10.4% of pages two through six, rising to 21.6% on Cloudflare-hosted subpages. That study counts a page as denied when it records a first-party 403 or 429, and states plainly that blocking conveyed through other status codes or through page content is not counted [5]. The two papers measure opposite halves of the same blind spot. One sees the refusals that announce themselves. The other sees the ones that do not.

Soft 404s: the site that answers every URL

13.1% of the sites whose random-path probe was answered answered it with 200 (10.5% for llms.txt sites, 13.3% for the Tranco top 10k, 17.3% for Markdown-serving sites, 13.7% and 13.1% for the lower bands) [1].

Every one of those is a soft 404 by construction, since the path was 16 random hexadecimal characters. But the bodies do not admit it. In an exploratory breakdown of 172 soft-404 responses, the labeler called 98 of them an error or substitute page, 32 a shell, and 28 ordinary content — a homepage or a listing served at a random path reads like a normal page [1].

Only 42 of them redirected to the root. But 133 of 170 had word 3-gram similarity of at least 0.8 with the same site's second random probe. The paper's conclusion is the useful one: a soft 404 is partly a property of the request–response pair rather than of the response alone. You cannot tell by looking at one page. You can tell by asking the same site for something that does not exist and comparing.

That is a two-request test any team can run this afternoon.

Rendering is not the cure

The obvious remedy for shells is to render. The paper measured that too, and the result has three edges.

Rendering helps where you expect. It gave main content for 75% of static shells, and it turned 43.5% of explicit static blocks into 200 responses [1]. Note what that second figure implies, though: those newly-200 responses are now exposed to exactly the 200-but-unusable failures this article is about.

Rendering fails in both directions. When the rendered page does have main content, the static response is still unusable for 11.4% of pages — shells, challenges served only to the static client, low retention. Going the other way, for 6.5% of pages whose static response is content, the rendered fetch is not, because rendering failed, returned no text, or the headless browser got a block page, a shell or a consent overlay. And the rendered reference is itself not type A for 16.7% of the static-200 items. A browser check by hand found the rendered client missed content a normal desktop browser showed for 3 of 47 pages, 6.4% [1].

Rendering is expensive. Median rendered fetch 19.7 seconds against 1.6 seconds static, roughly 12 times longer [1].

The paper's own summary is that outcome labeling remains necessary after rendering. Rendering changes which responses are wrong. It does not remove the need to check.

The measurement error this creates downstream

This is the part that should change how you read any AI-citation study, including the ones you commission.

The paper re-fetched the sources cited behind expert-verified claims in ExpertQA — 6,156 claim–URL pairs across 4,795 URLs and 3,383 sites, keeping only claims whose support was judged complete or partial with an evidence passage of at least 100 characters. A judge model compares the claim against the 2023 evidence passage and against the best-matching 4,000-character window of the page as it is now. A pair silently drifts when the original passage supported the claim and the live page, fetched with 200 and judged usable, no longer does [1].

11.0% of pairs point to dead URLs and 36.4% fail explicitly. Those are identical under every scheme. The 200s are where the schemes diverge.

QuantityCounting every 200 as contentUsing outcome labels
Pairs live and usable52.6%40.5%
Pairs live but unusablenot visible12.2%
Silent citation-drift rate21.9%5.8%

From Table 6 of the preprint, all cited sources [1].

The drift rate is off by a factor of 3.8. And the reason is mechanical rather than subtle: of the 274 apparent drifts under the status-only view, 217 — 79% — are responses labeled unusable, broken down as 96 error, substitute or parked pages, 73 block or challenge pages, and 48 shells or partial pages [1]. The judge was asked whether a shell still supports a claim. It said no. The study recorded that as the source having changed.

Dead-link counting carries the same error in the same direction. Counted by status, 11.8% of the cited URLs are dead; counting the 200 responses labeled error, substitute or parked as well, 16.0% no longer serve the cited content [1].

There is one more result here that deserves care, because it is the most quotable and the thinnest. Broken down by which answer system produced the citation, status-only drift runs about three times the label-based value for three of ExpertQA's four systems and five times for one. The paper flags its own limit: those label-based values rest on 6 to 26 drift events per system [1]. The ranking-relevant observation is sturdier than the numbers — sources retrieved from a static corpus were live and usable less often, so comparing answer systems by re-fetched support partly ranks them by the fetchability of their sources.

How to detect it cheaply

Detectors in the study judge the static response only, the way an agent would before deciding whether to render, retry or skip. Thresholds were fixed on a validation split at a false-alarm rate of at most 5%, and the splits are site-disjoint by registrable domain.

DetectorRecall on unusableFalse-alarm rate
Status code0.000.00
crawl4ai block check0.250.02
Visible text below a length threshold0.760.07
Random-path probe comparison0.350.03
Redirect from deep URL to root0.100.00
Challenge and block string matching0.060.02
Tuned combination of the above0.860.06
Zero-shot LLM0.570.06
TF-IDF text classifier0.910.13
Fine-tuned 1.7B classifier0.820.13

Reviewed test sample, 99 unusable and 195 usable items [1].

The winning combination is specific enough to implement from this sentence: a redirect to root, or probe similarity of at least 0.3, or fewer than 930 characters of visible text. The crawl4ai block check and the challenge-string matcher were switched off by the tuning, because they added false alarms without adding coverage.

Text length alone is the strongest single signal at 0.76, which tells you something about the shape of the problem: shells, challenges and block pages are short. By type, the tuned combination caught 11 of 12 challenges, 27 of 30 substitute or error items, and 39 of 48 shells. The 14 it missed are instructive — 9 were shells carrying enough navigation, SEO or license text to clear the length cut-off, the "Loading..." product grid and the sports page whose scores arrive by script [1].

The learned detectors are not worth the trade here. The TF-IDF classifier gets the highest recall at 0.91, but its false-alarm rate is 0.13 — more than twice the combination's — and 0.27 on usable cited sources. Its 26 false alarms included 14 ordinary articles with no visible defect and 5 pages that were short but complete, like a directory index and a plain-text file [1].

The practical payoff is the escalation policy, simulated on 920 test-split pages that returned a static 200.

PolicyUsable content obtainedUnusable passed on silentlyPages renderedMean seconds per page
Never render74.3%25.7%0%2.5
Always render80.3%0%100%20.6
Render only what the combination flags81.4%7.8%19.6%5.8

From Table 5 of the preprint [1].

Rendering a fifth of the pages beat rendering all of them on usable content obtained — 81.4% against 80.3%, because some pages are usable only statically — at 5.8 seconds per page instead of 20.6, while cutting silent failures from 25.7% to 7.8%.

What this does not tell you

The limits here are real and the paper states most of them itself.

One vantage point, three days. All fetches came from a single location between 19 and 21 October 2026. Block and challenge rates depend on the client, its IP address and the time, so the absolute share of each type belongs to that setup. The paper expects the same types to recur and makes no claim about the same levels [1].

The clients did not identify themselves as crawlers. Both sent the same desktop Chrome user-agent string. They obeyed the robots.txt rules that apply to all user agents under RFC 9309 [6], waited at least a second between requests to a host, never logged in, and never attempted to solve or bypass a challenge. One documented deviation: where robots.txt itself could not be reached, RFC 9309 says to assume a full disallow, and the study attempted the page once anyway so that dead hosts stayed countable [1]. What this means for you is that the measured block and challenge rates describe an unannounced browser-like client, not a declared AI crawler. Rates for GPTBot or OAI-SearchBot could differ in either direction, and nothing here measures them.

The drift gap includes judge error. The paper is explicit: the drift judge is an LLM applied under every scheme, and because each scheme hands it a different subset of pages, its errors need not cancel between schemes. So the gap between 21.9% and 5.8% may contain judge error alongside the effect of unusable pages [1].

No peer review. arXiv:2611.00884v1, single author, 10 pages, one version, no venue. The internal cross-checks are unusually careful for a preprint — a second model family reviewing blind, a human blind pass, a browser check of the rendered reference — and none of that substitutes for review.

And one thing nobody measured. Whether a shell, a soft 404 or a consent wall changes whether an AI engine cites your page is not in this paper and not in any source read for this article. The measurement is about what a fetch returns. The engines do not publish what they do with a fetch that returns nothing useful.

FAQ

Does a 200 with no content hurt my AI visibility?

No source read for this article answers that, in either direction. What the evidence supports is narrower and still useful: if your page answers automated fetches with a shell or a gate, an agent that fetched it did not read it, and any audit built on status codes will report the page as fine. Whether the engine then declines to cite you, cites you from a cached or third-party copy, or cites you anyway from a different fetch path is unmeasured here. Treat anyone quoting a citation-rate effect from this measurement as going beyond it.

Our pages are server-rendered. Are we exempt?

Shells are the largest failure group but not the only one, and the other six types have nothing to do with your rendering strategy. Soft 404s come from how your server answers URLs that do not exist — 13.1% of probed sites answered a random 16-character path with 200. Consent walls come from your privacy vendor. Substitute pages come from redirecting deep links to the homepage. Run the random-path probe against your own site before concluding anything from your rendering setup.

Is this just the old "AI crawlers don't execute JavaScript" argument?

It is the opposite direction of the same problem, and the distinction matters. That argument is about the client's capability. This measurement is about what the response contains regardless of client capability — and it found that rendering leaves 11.4% of pages unusable even when the rendered version has content, while breaking 6.5% of pages that were fine statically. Rendering changes the composition of the failures rather than eliminating them.

Should I stop blocking bots to fix this?

These are separate problems with separate signatures, and conflating them would be an expensive mistake. Your edge rules announce themselves: Cloudflare's WAF block, IP block, country challenge and managed challenge all return 403, and rate limiting returns 429 [3]. Those decisions are yours to make on security grounds and they are already visible to anyone checking status codes. The failures in this article are the ones your stack produces while reporting success — shells, soft 404s, embedded gates, placeholder pages. Fixing those does not require loosening anything.

What is the minimum version of this check?

Three requests and a character count. Fetch the page and record the length of the extracted text. Fetch a random 16-character path on the same host and see whether it returns 200. If it does, compare the real page against it. Flag anything that redirects a deep URL to the root, matches that probe closely, or comes back under roughly 930 characters — the threshold the paper's tuning selected [1]. That combination recovered 86% of unusable responses at a false-alarm rate near 5%, and it needs no model.

Bottom line

The status line was never a content signal, and Google says as much about its own crawlers: 2xx does not guarantee indexing, and an empty page or an error message behind a 200 shows up as a soft 404 [2]. What this measurement adds is the size of the gap — one in five successful fetches on the open web, more than one in four on the most popular sites — and the fact that it propagates into every number built on top of fetched pages [1].

Three things follow for anyone auditing their own site or reading someone else's citation study.

Stop accepting 200 as evidence that a page was read, in your own tooling and in reports you are shown. Ask for the outcome distribution: how many of those successful fetches were inspected, and what did they contain.

Run the two-request soft-404 probe against your own hosts. It is the single cheapest thing on this list and 13.1% of probed sites failed it.

And when a vendor shows you a drift rate, a dead-link rate or a "your content is accessible to AI" score, ask which status codes they counted. If the answer is 200 versus everything else, the number is off in a known direction by a known mechanism, and now you know roughly how much.

References

[1] - When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition (arXiv:2611.00884v1, preprint, not peer reviewed; Shuyang Zhang, The Hong Kong Polytechnic University, submitted 31 October 2026)
[2] - How HTTP status codes affect Google's crawlers — 2xx does not guarantee indexing; empty or error content behind a 200 is reported as a soft 404
[3] - Error page types — Cloudflare Rules documentation: WAF block, IP/country block, IP/country challenge and managed challenge return 403; rate limiting block returns 429 (last updated 16 April 2026)
[4] - Cloudflare Turnstile — embeddable into any website without sending traffic through Cloudflare, and usable without the Cloudflare CDN (last updated 14 August 2026)
[5] - Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls (arXiv:2611.06197v1, preprint, not peer reviewed; submitted 4 November 2026) — counts a page as denied only on a first-party 403 or 429
[6] - RFC 9309: Robots Exclusion Protocol — the robots.txt rules the study honored, including the unreachable-robots.txt case

Related Articles