AI bot & crawler traffic
Copy linkAI bots scan public websites to collect data used in search engines, AI model training, and other data processing tasks
Latest AI blog posts
Copy linkPosts tagged with "AI" from the Cloudflare blog
Google's dual-purpose crawler creates an unfair AI advantage. To protect publishers and foster competition, the UK’s Competition and Markets Authority must mandate crawler separation for search and AI.
Maria Palmieri, Sebastian Hufnagel
Moltworker is a middleware Worker and adapted scripts that allows running OpenClaw (formerly Moltbot, formerly Clawdbot) on Cloudflare's Sandbox SDK and our Developer Platform APIs. So you can self-host an AI personal assistant — without any new hardware.
Celso Martinho, Brian Brunner, Sid Chatterjee, Andreas Jansson
The Astro Technology Company team — the creators of the Astro web framework — is joining Cloudflare. We’re doubling down on making Astro the best framework for content-driven websites, today and in the years to come.
Fred Schott, Brendan Irvine-Broque