AI search may collapse the open web, Harvard study warns

Key Takeaways

• Google AI Overviews reduce organic clicks to publishers by 39.8% and increase zero-click searches by 34.5%
• AI answers destroy not just revenue but the quality signals (backlinks, subscriptions, repeat visits) that help users find trustworthy content
• Harvard researcher argues licensing payments are insufficient; the web needs provenance systems and exploration credits to survive

AI search is draining traffic from publishers at an alarming rate. A new field experiment shows Google AI Overviews cut outbound organic clicks by 39.8% while boosting zero-click searches by 34.5%. Harvard Business School researcher Alex Chan goes further, arguing in a paper titled "AI and the collapse of the www" that the damage extends beyond lost revenue. The web's information quality depends on signals like subscriptions, repeat visits, and backlinks. AI answers destroy those too.

What the research actually shows

The click data comes from Saharsh Agarwal at the Indian School of Business and Ananya Sen at Carnegie Mellon University. Their field experiment measured what happens when Google serves AI-generated answers directly on the search results page. The findings are stark: nearly 40% fewer clicks to source websites, with users getting their answers without ever leaving Google.

"Overall, the results suggest that AI Overviews divert traffic away from publishers without improving the user experience or quality of engagement for websites," the researchers conclude. Google CEO Sundar Pichai has disputed this narrative, but Pew research supports the opposite view.

Chan's Harvard paper asks a different question: what happens after the traffic disappears? His answer is troubling. The open web runs on a bargain. Publishers create content. Search engines send visitors. Those visits generate revenue and something equally important: quality signals.

Why lost traffic is only half the problem

When someone visits a website, subscribes to a newsletter, bookmarks a page, or links to it from their own site, they create information. That information tells search engines and future visitors which sources are trustworthy. Chan calls this "durable attention capital." It includes subscribers, repeat readers, backlinks, reputation, and search authority.

AI answers intercept all of it. The user gets their answer inside ChatGPT or Google's interface. The source website gets nothing. No visit, no subscription opportunity, no backlink, no signal that their content was valuable. Over time, this makes it impossible to distinguish quality journalism from AI-generated slop.

"When an AI platform diverts the revenue and measurement events without replacing them, costly human information may fall below replacement," Chan writes. In plain terms: creating good content becomes economically irrational. The people who do it will stop.

Why licensing payments won't fix this

Some AI companies have signed licensing deals with publishers. OpenAI pays The Atlantic and Axel Springer. Google has similar arrangements. Chan argues these "visitor replacement royalties" miss the point.

The problem is not just that publishers lose money. Licensing deals address that, at least for big publishers who can negotiate them. The deeper problem is the collapse of quality signals. A licensing payment does not generate a backlink. It does not create a subscriber relationship. It does not tell other users that this source is worth reading.

Chan also rejects banning AI answers outright. That would prop up the old model instead of building a new one. His proposed solutions sound technical: provenance systems, diversity prices, exploration credits, and informative audits. The goal is to create new quality signals that work even when users never leave the AI interface.

The slop arbitrage problem

There is a catch. Distinguishing human-authored content from AI-generated content is hard. It may get harder. And there is profit in selling AI slop as premium human work.

Chan acknowledges this. Any system that rewards human-created content creates an incentive to fake it. Expect "active resistance to labeling or other mechanisms that threaten slop arbitrage," as The Register puts it. The same market dynamics that make AI answers profitable make AI content laundering profitable too.

Companies like Cloudflare are already trying to address part of this. They recently announced plans to block AI scrapers that take content without forwarding traffic. But blocking scrapers is a defensive move. It does not solve the fundamental economic problem Chan identifies.

What this means for content businesses

Publishers who depend on search traffic face an uncomfortable choice. They can negotiate licensing deals with AI companies, accepting a one-time payment for content that previously generated recurring visits. They can block AI crawlers and hope enough competitors do the same to starve the models of training data. Or they can build direct relationships with readers through newsletters, memberships, and communities that do not depend on search discovery at all.

The third option is the hardest but may be the most durable. Email newsletters through platforms like Kit or Mailchimp create subscriber relationships that no AI answer can intercept. The reader's inbox is still territory you control.

Frequently Asked Questions

What is a zero-click search?

A zero-click search occurs when a user's query is answered directly on the search results page, so they never click through to a source website. AI Overviews have increased these by 34.5% according to recent research.

How much traffic have publishers lost to AI Overviews?

A field experiment by researchers at Carnegie Mellon and the Indian School of Business found AI Overviews reduce outbound organic clicks by 39.8%.

Why won't licensing deals solve the problem?

Licensing payments compensate for lost revenue but do not replace the quality signals (backlinks, subscriptions, repeat visits) that help users and search engines identify trustworthy sources.

What does 'durable attention capital' mean?

A term from Harvard researcher Alex Chan referring to the long-term value created by user interactions: subscribers, repeat readers, backlinks, bookmarks, and search authority. AI answers intercept these signals.

Can publishers block AI from scraping their content?

Yes, through robots.txt directives and services like Cloudflare's new anti-scraper tools. However, blocking does not solve the economic problem if competitors do not follow.

Source: www.theregister.com