The symbiotic bond between online content creators and search engines is under threat, warns Cloudflare CEO Matthew Prince. This shift isn’t just a minor hiccup—it could destabilise the entire internet economy. Prince points to a sharp drop in web traffic and monetisation for publishers, blaming factors like Google, emerging AI tools, and what he calls ‘naive’ licensing agreements.
Just a decade ago, Cloudflare noted a 2:1 ratio of pages crawled by Google compared to visitors sent back to the original sites. Today, that figure has ballooned to 18:1 as Google increasingly serves answers directly on its platform. It’s been reported that 75% of searches now end without a click through to another site, a number that might soon hit 90% with the introduction of Google’s new ‘AI Overview’ feature.
AI search engines are making matters worse. For instance, OpenAI recently crawled 1,500 pages for every visitor it directed to a source—a dramatic jump from its earlier 250:1 ratio. Anthropic’s numbers are even steeper at 60,000:1. As users rely on AI-generated summaries without checking original sources, publishers see their reach and revenue shrink. This trend not only disrupts individual business models but also chips away at the foundation of a free and open internet.
Prince is quick to warn about the risks of overly permissive licensing agreements with AI companies such as OpenAI. Some publishers, in an effort to secure quick cash, allow unrestricted access to their content. “You can’t have a market if you don’t have scarcity,” he explains. While these deals might offer short-term relief, they can ultimately devalue the content. As Prince puts it, “The renewal of the deal that you signed today will be worse tomorrow, I guarantee that.” Even journalism professor Jeff Jarvis has criticised these payments as little more than ‘pure lobbying,’ trapping the industry in what looks like a tech-driven prisoner’s dilemma.
To combat this, Cloudflare is proposing a collective block against AI crawlers—a measure aimed at restoring scarcity. The company is set to launch its protection system in New York by the end of June, inviting publishers to band together and activate the block. This initiative has already sparked interest among major media houses and some AI providers.
Looking ahead, Prince imagines a future where content is valued for its contribution to knowledge rather than just page views. He envisions scenarios where large language models licence specialised information—say, in medical fields—instead of merely tracking clicks. “Imagine a time when you’re rewarded for advancing human knowledge rather than just driving up page views,” he suggests. “If we don’t fix this, the internet will die.”
