From Signed Identity to Trust Bands: Cloudflare Starts Scoring Whether AI Crawlers Respect Your Site

A driver's license proves who someone is and that they once passed a test. It says nothing about how they will drive tomorrow. Cloudflare opened a post on October 16, 2026 with roughly that comparison, and then drew the conclusion: verifying a crawler's identity creates accountability, but trust depends on what it does next [1].

That post introduced a system that observes whether a Verified bot's behavior starts to drift from the rules it agreed to, and turns those observations into what Cloudflare calls a trust band [1].

The identity half of this problem was solved first. Cloudflare's Verified bots program now accepts cryptographic request signing through Web Bot Auth, built on HTTP Message Signatures (RFC 9421), so a crawler can prove it is who it claims rather than merely assert it in a header [3][4]. The behavior half is what October 16 addressed.

For anyone running a site, this shifts the question. The common GEO instruction has been to allow search crawlers and block training crawlers by name in robots.txt. That instruction operates on strings. The infrastructure underneath is moving to proof-based identity plus measured conduct, and two of the signals being measured are produced by your server: your robots.txt and your HTTP 429 responses.

---

The Short Answer

Cloudflare's Verified bot program now rests on two bars rather than one: honest self-identification, and non-abusive behavior [2].

1. Identity became cryptographic. Web Bot Auth has a crawler sign each request with a private key and publish the public half at a well-known URL. Cloudflare validates the signature at its edge and marks the traffic verified, which site owners can then match in firewall and bot rules [3][4].
2. Behavior is now measured over time. The new system watches two starting signals: whether a bot's traffic stays consistent with the preferences a site publishes, and whether it responds to direct feedback. Those observations become an explainable trust band, described as a range with the behaviors behind it rather than a permanent verdict on an operator [1].
3. Your server emits the evidence. robots.txt is the primary channel for publishing preferences, and an HTTP 429 is the clearest way to tell a crawler to slow down. The system looks at whether traffic adapts after a 429: slowing or pausing builds trust, repeatedly ignoring it lowers the band [1].

What has not happened: the scores are not public. Cloudflare says it is using the system internally for now, plans to publish on Radar later, and plans to let customers pick the trust threshold that decides how Verified bot traffic is handled [1]. None of this is an industry standard, and no vendor has claimed a trust band affects whether an AI platform cites you.

The practical takeaway is narrower than the headline and more useful: the signals your origin sends are becoming inputs to somebody else's measurement, so they are worth getting right.

---

Why Identity Had to Be Rebuilt First

Cloudflare's own account of the old method is unflattering. Verification leaned on two signals: the User-Agent header and published IP ranges. A User-Agent is a string the client chooses, so anyone can set it. IP ranges are brittle for a different reason: the same addresses can be shared across users, across services inside one company, or across companies on shared hosting, and the ranges shift as infrastructure changes [3].

The alternative Cloudflare proposed, and has since folded into the Verified bots program, is to make the crawler prove its identity per request.

How a signed request is put together

The signing side works like this [3][4]:

• The crawler generates a signing key and hosts a JSON Web Key Set containing the public half at /.well-known/http-message-signature-directory on its own domain.
• It registers that directory URL with Cloudflare through the Verified Bots form, selecting request signature as the verification method.
• On each outgoing request it signs the authority of the target URI, writing a Signature-Input header that carries a validity window (created and expires), a keyid that is a JSON Web Key Thumbprint, and a tag set to web-bot-auth.
• A Signature-Agent header points to the key directory, and that header is itself part of the signed content.

A simplified example of the headers, following Cloudflare's published shape:

GET /path/to/resource HTTP/1.1
Host: www.example.com
Signature-Agent: "https://signer.example.com"
Signature-Input: sig=("@authority" "signature-agent");\
                 created=1700000000;\
                 expires=1700011111;\
                 keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";\
                 tag="web-bot-auth"
Signature: sig=:jdq0SqOwHdyHr9...:

How the edge checks it

Cloudflare's verification runs as an ordered set of checks [4]:

Two details matter for site owners. First, site owners need no configuration: Cloudflare validates signatures at the edge and marks successful traffic as verified, which can then be matched in Bot Management and WAF rules [4]. Second, when any check fails, Cloudflare falls back to its older identification methods. Cloudflare describes this as a soft downgrade it intends to narrow as the system matures [4]. Signed identity is therefore an upgrade in confidence, not yet a hard gate.

Signing is also not confined to Cloudflare. RFC 9421 is a published standard, and Cloudflare notes that OpenAI signs Operator requests with it. Web Bot Auth itself is draft work at the IETF, and as of Cloudflare's post the IETF was considering forming a working group around it [3][4].

---

What the October 16 System Actually Measures

Cloudflare frames the Verified bot bargain as three obligations: identify yourself accurately and securely, do only what you say you will do, and do not behave abusively [1]. Signing covers the first. The new system goes after the rest.

Risk and trust are treated as different quantities

The post separates two ideas that get conflated. Risk is the chance that one request or action is harmful, and it can change moment to moment. Trust accumulates through repeated, consistent behavior [1]. Cloudflare had set out the same split earlier, arguing that one-time hurdles such as CAPTCHAs are risk instruments because they carry no history, while behavioral assessment across a session is a trust instrument [5].

That distinction is why the output is a band rather than a pass or fail. Cloudflare describes the trust band as a range showing how consistently a bot respects published preferences, accompanied by the observed behaviors that produced the assessment, and explicitly not a permanent judgment about a bot or its operator [1].

The two starting signals

Both are drawn directly from the post [1]. Cloudflare also makes the point that a crawler can stay entirely inside allowed paths and still put heavy load on a site, which is why rate behavior is measured separately from path compliance.

There is precedent for consequences. Cloudflare removed Perplexity's bot from its Verified list in 2025 after finding it reached resources on sites that had opted out of scraping and then tried to work around the blocks [1]. Its report at the time described repeated user-agent changes, shifting source ASNs, and robots.txt files that were ignored or not even fetched [6].

Current status, stated plainly

Cloudflare says it is already running this internally to assess how consistently Verified bots honor site preferences, and is using the initial phase to build history and refine how observations become guidance. Publishing scores on Radar, broadening the measurements, and exposing a customer-selectable trust threshold are all described as plans [1].

So: real system, real precedent, not yet a dial you can turn.

---

The Sentence Site Owners Should Read Twice

Buried in the same post is a statement about robots.txt that contradicts how many teams use the file. Cloudflare writes that robots.txt sets the minimum of what bots should not do, and that nothing follows about whatever the file does not mention being permitted. Its example: the file does not need to state that a bot must stop, because a site can signal that continued access is unwelcome through other channels [1].

Two consequences follow.

If you maintain a careful allowlist in robots.txt and assume silence equals consent for everything else, that reading is not the one the measuring party holds. And if your edge rules say something different from your robots.txt, you are publishing two preferences at once. The second case is common enough that Cloudflare built a feature for it, covered below.

It is also worth keeping the file's actual force in view. Cloudflare's own documentation states that robots.txt compliance is voluntary, that the file expresses preferences without technically preventing access, and that some operators will crawl regardless [7]. The trust band does not change that. It changes whether disregarding the file leaves a durable record.

---

403 Versus 429: The Most Actionable Detail Here

If a crawler is overloading your origin, the status code you answer with now does more than stop the request. It determines whether you communicated a rate instruction at all, and whether any adaptation can be observed.

Google documents its side of this explicitly. To reduce crawl rate urgently for a short window, measured in hours or one to two days, return 500, 503, or 429 instead of 200. When Google's infrastructure sees a significant number of those codes, it reduces the crawl rate across the whole hostname, covering both the erroring URLs and the ones still serving content, and raises the rate again once the errors subside [8].

With 503 or 429, Google also reads a Retry-After header as defined in RFC 9110, accepting either a delay in seconds or an absolute UTC timestamp [8]:

HTTP/1.1 429 Too Many Requests
Retry-After: 120

HTTP/1.1 503 Service Unavailable
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Google restructured this guidance and added the Retry-After material on October 9, 2026, noting that support for the header was not new but had been documented elsewhere and was hard to find [9].

Put the two sources together and a practical rule appears:

A team that answers every crawler problem with 403 gets the traffic to stop and generates no record that it ever asked for a slowdown. A team that answers with 429 and Retry-After both communicates the limit and produces the evidence the measurement depends on.

Two cautions. Google's error-code method is framed for short emergencies, not as a standing configuration, and it suppresses crawling across the entire hostname while it is in effect [8]. And Cloudflare's measurement covers Verified bots on its network; it says nothing about how an unverified or non-compliant crawler will react.

---

Making Your Published Preferences Match Your Enforcement

Cloudflare's Bot Preference Sync exists because the two layers drift apart. It generates managed robots.txt directives from the Search, Agent, and Training policies set in the dashboard, and it only writes them when at least one policy is set to Block, Block on pages with ads, or, for Training, Disallow [7].

Mechanics worth knowing before turning it on [7]:

• If your origin already serves a robots.txt, confirmed by an HTTP 200, Cloudflare prepends its managed block ahead of your existing content and serves both in one response. If you have no file, it creates one.
• Setting Training to Disallow adds the managed directives and turns on edge blocking for AI training crawlers. Mixed-use crawlers are excluded from that edge block. A crawler that does both search and training will not be stopped by it.
• The managed block includes Content Signals, a machine-readable vocabulary with three fields: search for building a search index and returning links and short excerpts, ai-input for feeding content into models for real-time answers such as retrieval-augmented generation, and ai-train for training or fine-tuning.
• Cloudflare is testing a fourth field, content-use, describing what may be kept and reused after access, with three values: use=immediate to interact but store nothing, use=reference to index, excerpt, and link back, and use=full to summarize and reproduce. Managed content currently carries use=reference.

The resulting managed block looks like this in outline:

# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content

The mixed-use exclusion deserves emphasis because it interacts with the separate control Cloudflare shipped in September. That Disallow AI Training setting was built specifically so a site could stay indexed for search while refusing training by the same crawler, and Cloudflare said Apple, Google, and Microsoft honor it or have committed to doing so within a stated window [12]. Separately, new domains onboarding to Cloudflare from September 15, 2026 receive defaults that block Training and Agent behavior on pages displaying ads while leaving Search allowed [11].

If you have never audited which of these applies to your zone, the file you publish and the rules you enforce may already disagree.

---

Behavior Labels Are Now More Precise Than User Agents

The other reason to stop writing rules against user-agent strings is that Cloudflare's classification has become considerably more granular. A single bot can carry more than one behavior label [2]:

Search, Agent, and Training are available as managed presets on all plans. Site owners can filter on cf.verified_bot_category inside WAF custom rules, advanced rate limiting, and late transform rules, which is a more stable hook than a user-agent match [2][4].

Cloudflare also labels how a bot is operated. Direct means a single narrow operator on its own infrastructure. Intermediary means an agentic service that many different end users drive. The second case creates transitive trust: you may trust the operator while knowing nothing about the individual end user behind a given request. Cloudflare says it is experimenting with forwarding end-user information using the Forwarded header from RFC 7239 so that site owners can apply preferences to the responsible party [2].

BotBase, the directory behind all this, is deliberately not a list of approved bots. Cloudflare describes it as a directory of all known bots and agents, capable of tracking poorly behaved ones, and states that abusing trust on its network should cost a bot its verified status [5][10].

---

What This Changes for GEO, and What It Does Not

The last row matters most for anyone selling or buying GEO work. Access is a precondition, not an outcome. A crawler reaching your page is followed by indexing, retrieval, reranking, evidence selection, and generation before anything appears in an answer, and none of those steps is governed by a trust band. Google's own guidance for its generative AI features puts crawlable content as a requirement for eligibility and says nothing about crawler scoring.

What genuinely changes is the weight of your own signals. Before, a disregarded robots.txt was an annoyance with no consequence. Now it is an input to a measurement that Cloudflare intends to publish and to expose as a customer-configurable threshold [1]. That is a reason to make your published preferences accurate, not a reason to expect new traffic.

---

A Seven-Step Audit

Step 1: Capture what you actually serve

Fetch robots.txt from every variant that resolves: apex and www, HTTP and HTTPS, and any regional hostnames. Cloudflare's community forum carries reports of managed directives not appearing as expected on some configurations, so verify the served response rather than the dashboard setting.

Step 2: Capture what you actually enforce

Inventory every layer that can answer a crawler: Cloudflare bot policies, WAF custom rules, rate limiting, origin rules, and anything in front of or inside your application. The relevant question for each is which status code a crawler receives and under what condition.

Step 3: Reconcile them

Line the two lists up and mark every disagreement. A robots.txt that permits a crawler which your WAF blocks with 403, or a managed Disallow for a crawler your rules happily serve, are both published inconsistencies. If you use Bot Preference Sync, confirm which directives it is writing on your behalf [7].

Step 4: Fix the rate responses

Replace blanket 403 responses to crawler load with 429 plus Retry-After. Keep 503 with Retry-After for maintenance. Treat Google's error-code approach as a short emergency measure, since it suppresses crawl rate across the whole hostname while active [8].

Step 5: Decide the three behaviors separately

Search, Agent, and Training are different uses and deserve separate decisions. Write down what you allow for each, and record explicitly how you handle mixed-use crawlers, since the Training edge block does not cover them [7][12].

Step 6: Move off user-agent matching

Where your platform supports it, key rules on verified identity and behavior category rather than a string [2]. Keep user-agent rules only as a fallback, and know they can be set by anyone.

Step 7: Keep watching the answers

Access controls govern whether a crawler can reach you. They do not tell you what AI platforms say about you. That needs separate observation.

---

Where Innflows Fits

Everything above is infrastructure: who may fetch your pages, and whether they behaved. None of it reports what AI platforms actually tell users about your brand.

Innflows monitors AI answers, citations, and brand mentions across major AI platforms and languages. Around a crawler-policy change, a team can use it to:

• record which of its pages AI answers cite before a policy change, as a baseline;
• watch whether citations to a given set of pages change after Training, Agent, or Search policies are adjusted;
• check whether AI answers about the brand stay accurate when access rules tighten;
• compare how different platforms describe the brand once a policy is in place.

The boundaries are firm. Innflows does not read CDN or server logs, does not see Cloudflare trust bands or Verified bot status, and cannot tell you whether a specific crawler obeyed your robots.txt. It observes public AI answers from the outside. A change in citations after a policy change is a correlation across runs, not proof that the policy caused it, and no monitoring tool can guarantee citations.

---

Six Common Misreadings

1. "My site gets a trust score now."

The band scores crawlers. Cloudflare's system assesses Verified bots; your site is the source of signals, not the subject of the assessment [1].

2. "Signed identity means spoofing is over."

When a signature check fails, Cloudflare falls back to its older methods, a soft downgrade it says it intends to narrow over time [4].

3. "A trust band will affect whether AI cites me."

No source claims that. Citation depends on retrieval and generation steps that sit well downstream of access.

4. "Anything robots.txt does not forbid is allowed."

Cloudflare states the opposite: the file sets a minimum, and its silence grants nothing [1].

5. "Blocking Training in Cloudflare stops all training crawling."

Mixed-use crawlers are excluded from the Training edge block [7]. The separate Disallow AI Training setting was built for exactly that case [12].

6. "This is how the whole web now works."

It is one network's mechanism. Web Bot Auth is IETF draft work, RFC 9421 is the published piece underneath it, and the trust band is Cloudflare's own and still internal [1][3].

---

Frequently Asked Questions

What exactly did Cloudflare announce on October 16, 2026?

A system that evaluates and reports when a Verified bot's behavior starts to deviate from the program's rules, producing an explainable trust band from observed conduct. It is running internally, with plans to publish on Radar and to let customers choose a trust threshold [1].

What is Web Bot Auth?

A method for a crawler to prove its identity by cryptographically signing requests, using HTTP Message Signatures from RFC 9421. Cloudflare validates the signature at its edge; the crawler publishes its public key at a well-known directory on its own domain [3][4].

Do I have to do anything to benefit from signature verification?

No. Cloudflare validates signatures at the edge and marks successful traffic as verified, which you can then reference in bot and firewall rules [4].

Should I use 403 or 429 for a crawler that is overloading my site?

Use 429 with Retry-After when you want the crawler to slow down but still have access. A 403 conveys refusal and no rate instruction. Google documents reducing crawl rate on 500, 503, or 429, and reads Retry-After on 503 and 429 [8].

Does a higher trust band mean more AI visibility?

There is no evidence for that. The band concerns whether a crawler respects site preferences, not whether an AI platform cites a page.

Has a crawler ever lost verified status over behavior?

Yes. Cloudflare de-listed Perplexity's bot in 2025 after reporting that it accessed resources on sites that had opted out and then attempted to circumvent the blocks [1][6].

Is robots.txt enforceable now?

No. Cloudflare's documentation still describes compliance as voluntary and notes that some operators crawl regardless [7]. What changed is that disregard can now accumulate into a record.

---

The Bottom Line

The interesting part of the October 16 announcement is not the scoring. It is the admission behind it: proving who a crawler is turned out to be the easy half, and it does not answer the question site owners actually have, which is whether the crawler will behave.

Cloudflare's answer is to measure conduct over time, starting with two signals your own server produces. That gives your robots.txt and your 429 responses a weight they did not have before, and it makes contradictions between what you publish and what you enforce a liability rather than a quirk.

None of it moves you closer to being cited. Access was always the precondition, never the outcome.

Get the signals you control right, then go measure the thing you actually care about: what the answers say about you.

---

References

[1] - Continuous accountability for Verified bots — Cloudflare, October 16, 2026

[2] - Verified bots — Cloudflare bot solutions documentation

[3] - Forget IPs: using cryptography to verify bot and agent traffic — Cloudflare

[4] - Message Signatures are now part of our Verified Bots Program — Cloudflare

[5] - Unveiling good and bad behaviors on the Agentic Internet — Cloudflare

[6] - Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — Cloudflare, August 2025

[7] - robots.txt setting and Bot Preference Sync — Cloudflare bot solutions documentation

[8] - Reduce the Google crawl rate — Google Search Central

[9] - Google's Crawling Documentation Changelog — Google, October 9, 2026 entry

[10] - BotBase — Cloudflare bot solutions documentation

[11] - New options to manage AI traffic — Cloudflare changelog

[12] - Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare, September 2026