SOURCE

Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Properties1
tagsagent-web, monetization, primary-source, cloudflare

2 min read

Introducing pay per crawl (Cloudflare Blog)

Summary

Cloudflare’s official launch of Pay per Crawl, letting publishers charge AI crawlers instead of choosing only between allow and block. Publishers set one flat per-request price site-wide and configure each crawler as Allow / Charge / Block; negotiated partnerships can bypass the system.

Technically it works over standard HTTP: a paying crawl returns 200 OK with a crawler-charged header; an unpaid one gets 402 Payment Required with a crawler-price header. Two flows exist: reactive (get the 402, retry with crawler-exact-price) and proactive (send crawler-max-price upfront). Crawler identity is anti-spoofed via Web Bot Auth — Ed25519 key pairs and HTTP Message Signatures (RFC 9421). Cloudflare acts as Merchant of Record, aggregating billing and paying out publishers.

Status at publication: private beta, explicitly early-stage. The stated future vision includes dynamic per-path pricing, licensing granularity (training vs. inference vs. search), and agents that programmatically negotiate and budget for content.

Key Claims

  • Publishers get three per-crawler options: Allow, Charge, Block; one flat per-request price across the site.
  • 402 + crawler-price / 200 + crawler-charged headers implement the price signal.
  • Crawler auth uses Ed25519 keys and RFC 9421 HTTP Message Signatures (Web Bot Auth).
  • Cloudflare is Merchant of Record — publishers don’t build payment plumbing.
  • Private beta only; flat pricing only (no dynamic/per-path pricing yet).
  • Agent-negotiated content marketplaces are the stated end-goal, not the shipped product.

Connections

Contradictions