Making AI search smarter

Author: Cloudflare | Published: 2026-07-01 | Source: article

Summary

A year after Content Independence Day, Cloudflare argues the original web bargain — let a search engine crawl you, get visitors, monetize the visit — is broken because answer engines summarize the page so the visit never happens. The supporting third-party number is the post’s most durable contribution to this wiki: a 2025 Pew Research Center study found that when Google shows an AI summary, users clicked a traditional result link 8% of the time (about half as often as when there is no summary) and clicked a link inside the summary only 1% of the time. Combined with Cloudflare Radar’s claim that more than 50% of online traffic is already non-human, publishers are stuck choosing between opting out of AI (and becoming hard to find) or opting in (and giving away the value). Blocking was step one; this post is the “say yes, but get paid and crawled less” program.

Make AI search smarter (research program). More than 20% of the web sits behind Cloudflare. The company will explore combining customer-shared freshness signals (“this page has/hasn’t changed”) with its own view of human and bot traffic, then offer that as a roadmap to answer engines: recrawl only what changed, surface what is actually being asked for. Cloudflare data suggests more than 50% of crawl traffic from good bots is re-fetching unchanged pages, a share expected to climb. A “nothing changed” signal saves the engine compute and saves the site owner from paying to serve empty recrawls. The program is scoped to search, shares no content, and is claimed not to train foundation models. Broad availability is planned later in 2026.

Pay creators (Pay Per Crawl → Pay Per Use). Pay Per Crawl (2025) charged AI companies per crawl, a crude value metric: one fetch can be cited in thousands of answers, or fetched forever and never used. Experiments with Ceramic.ai and You.com test paying on the outcome instead. Ceramic’s “pay-per-query” model pays opted-in publishers when their content appears in Ceramic’s results. You.com lets agents pay on demand for a specific premium asset with no upfront commitment. Other models (pay per query, pay per result) are in test. Participating Cloudflare/Ceramic publishers also unlock AEO reporting: top queries, the specific page and snippet, average ranking position. Cloudflare calls this an experiment at Internet scale, not a finished market.

The strategic reframe: the old web optimized search to save a human time (ten blue links, a click). The agentic Internet searches continuously — dozens of retrievals per question — so the unit that matters is the outcome, not the crawl or the click. Companion post the same day (Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402) generalizes charging from crawlers to any resource via x402; this post is the search-specific, creator-compensation half.

Key Claims

Pew 2025: when Google shows an AI summary, users click a traditional result 8% of the time (≈ half of no-summary) and click a link inside the summary 1% of the time. (third-party study, cited — highest-value datapoint in the post)
>50% of online traffic is non-human (Cloudflare Radar); >50% of good-bot crawl traffic is re-fetching unchanged pages. (Cloudflare data)
Publishers currently face a bind: opt out of AI and become unfindable, or opt in and give the value away. (vendor framing)
Freshness-signal research program: customer-shared change signals + Cloudflare traffic insight → answer engines recrawl less and surface fresher content. Search-only; no content sharing; not for foundation-model training. Broad availability planned later in 2026. (vendor program, not yet results)
Pay Per Crawl is too crude; experiments with Ceramic.ai (pay-per-query when content appears in results) and You.com (on-demand pay for a specific asset) are shaping Pay Per Use. (experiment, not GA)
AEO reporting for Ceramic program participants: top queries, page + snippet, average ranking position. Positioned as the first of several discoverability products. (product claim)
In an agentic world the unit of value is the outcome, not the crawl or the click. (vendor thesis)

Notable quotes

“When Google shows an AI summary, users clicked on a traditional search result link just 8% of the time (about half as often as when there’s no summary) and clicked a link inside the summary only 1% of the time.”

Why citable: Independent (Pew 2025) quantification of AI-overview click suppression. The wiki’s previous number was Pagepro’s vendor-ish “AI Overviews steal 30–70% of clicks”; Pew is the better citation. Use in client conversations about why GEO / AEO (Getting Recommended by AI) is not optional.

“A single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all.”

Why citable: The reason Pay Per Crawl is the wrong meter. Frames the shift from crawl-based to use/outcome-based compensation.

“The unit that matters isn’t the crawl or the click. It’s the outcome.”

Why citable: One-line economic thesis of agentic search. Pairs with the Monetization Gateway’s “the request becomes the transaction.”

Connections

Entities mentioned: Cloudflare, Google · Ceramic.ai, You.com, Pew Research Center (minor) · Pay Per Crawl (NEW: pay-per-crawl) Concepts referenced: GEO / AEO (Getting Recommended by AI), Technical SEO from Architecture

Contradictions / Tensions

Click-suppression numbers. Pagepro (Web Development Best Practices 2026: Engineering Guide) said AI Overviews/rich results steal 30–70% of clicks. Pew as cited here is more precise: 8% CTR with a summary vs. ~16% without (half), plus 1% on in-summary links. Those are not the same statistic (absolute CTR vs. “share of clicks stolen”) but they point the same direction. Prefer Pew for citation; keep Pagepro as vendor corroboration.
Dooley’s “AI leads tiny, branded search is the payoff” (2026 SEO Secrets That Ranked 1,000+ Sites (James Dooley, interviewed by Sterling Sky): ~1.3–1.4% direct LLM leads, 15–20% branded search) is compatible with Pew’s 8%/1% — people don’t click the summary, they come back later via brand. Cloudflare’s answer is get paid for the unpaid use; Dooley’s is capture the branded-search leftover. Both can be true; they are different business responses to the same CTR collapse.
Freshness recrawl vs. Google’s own crawl. Official SEO Starter Guide: The Basics already says directory-level change frequency helps Google crawl smarter, and that changes take weeks. Cloudflare is proposing it as the freshness oracle for all answer engines. That is both a web-dev cost win (less origin load) and a potential SEO risk if Googlebot is treated like any other bot and throttled. The post says the program is for answer engines willing to play fair; it does not say Google is a participant.
Pay Per Use is an experiment. Do not treat Ceramic/You.com as the market. Cloudflare says so itself.
Same-day tension with the Gateway post. Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402 says charge any resource, which can hide content from crawlers. This post wants to be findable and paid. The intended synthesis is: remain crawlable to paying/fair engines, gate the rest, get AEO reporting on the paid path. Unresolved if Google never pays.

Notes

Primary third-party citation for AI-overview CTR (Pew 2025). Primary source for Cloudflare’s AEO-reporting product and the Pay Per Crawl → Pay Per Use shift. Pair with Announcing the Monetization Gateway: charge for any resource behind Cloudflare via x402.

NEW pages to create (not written this ingest): pay-per-crawl (concept/product — 2025 charge-crawlers; this post evolves it to Pay Per Use), x402 (from the companion post). AEO reporting can aggregate into existing GEO / AEO (Getting Recommended by AI) rather than a new page.