SOURCE

The next step for content creators: Introducing AI Crawl Control

Properties1
tagsagent-web, primary-source, cloudflare

1 min read

Introducing AI Crawl Control (Cloudflare Blog)

Summary

AI Crawl Control is the GA rebrand of Cloudflare’s beta “AI Audit” tool: dashboards of AI crawler activity, per-bot selective blocking, and — for paid plans — customizable 402 responses with a message (e.g., “email [email protected]” or a pointer to a licensing API), creating a communication channel between site owners and crawler operators. Cloudflare notes customers already send over one billion 402 responses on an average day. Roadmap: structured parameters in 402 responses for content value, freshness, and licensing terms. Pay per Crawl remained private beta at this date.

Key Claims

  • GA as of 2025-08-28, paid Cloudflare plans only (free tier excluded).
  • Custom-message 402 blocking turns a block into a negotiation channel.
  • 1B+ 402 responses/day already flowing through Cloudflare customers.
  • Structured licensing metadata in 402s is roadmap, not shipped.

Connections

Contradictions

  • None. Matches the podcast’s “AI crawl control” description; adds availability detail (GA, paid plans).