The internet’s oldest bargain is breaking.
For decades, websites welcomed crawlers. Getting scraped by Google or any search engine meant getting indexed, ranked, discovered. For every two bots it sent to a website, Google sent one user. So, scraping meant web traffic and business.
But now, as generative artificial intelligence tools race to ingest the entire internet, scraping means getting bypassed. According to Cloudflare, for every user that ChatGPT maker OpenAI sends to a site, it sends 1,500 bots. For Anthropic, that figure is 60,000. Traffic from human visits is flattening while automated traffic surges, sometimes outnumbering people entirely.
And unlike the Google era, these bots usually don’t link back to their source material. Instead, AI models summarise and serve answers inside their own interfaces, keeping users locked in and cutting websites and content creators out of the loop.
Which goes to the heart of the issue. While much of the public conversation has focused on fears that AI will replace human workers, a more immediate threat isn’t attracting as much attention, even as it’s upending the underlying economics of the internet itself.
Bots have been crawling the web since the early 1990s — first to map it, then to index it, and eventually to fuel the rise of search. Early crawlers like World Wide Web Wanderer laid the groundwork for an era where visibility meant viability. By the time Google’s own crawler — Backrub, later Googlebot — came online in 1996, a new logic had taken hold: Let search engines scan your content, and in return, they’ll send you traffic.
For most of the next 30 years, that logic defined the open web. Visibility in Google results was everything. From niche blogs to the planet’s biggest media conglomerates, websites lived or died by their search ranking. Entire industries, not to mention consulting practices, sprang up around search engine optimization (SEO), information architecture, and content strategy, all to help businesses play the game better.
But AI doesn’t play by the old rules.
Instead of linking back to source material, LLMs like OpenAI’s ChatGPT, Anthropic’s Claude, and even Google’s own Gemini read and repurpose it — answering user queries directly, mostly without any attribution at all.
This shift is giving rise to a new acronym, AEO (meaning AI engine optimization), plus entire strategies for making content more visible to AI and more effective once absorbed inside it, even if AI answers never lead to clicks. If SEO shaped the search era, AEO is likely to define the age of generative AI.
AI bots, Tong said, are “synthesizing what they can access and crawl, and surfacing it in a format that strips away the site’s value proposition.” No clickthrough. No credit. Just answers.
Tong said Webflow has seen AI crawler traffic rise more than 125% in just six months. Across the broader internet ecosystem, more than 50% of all internet traffic now comes from bots. And as bot traffic soars, Tong said, some companies are starting to draw a line — literally. They’re building two versions of their websites. One is designed for humans, with visuals, interaction, and branded
storytelling. And the other is stripped down, optimized for machine readability, and engineered to feed the AI beast without giving away (so to speak) the crown jewels.
“For a human, your site should be rich, interactive, delightful,” Tong said. “For a bot? You want clear structure, easy crawlability, but maybe not your full content.” Some publishers are now exposing just a summary or excerpt to crawlers, hoping to bait the index without cannibalizing their monetization model.
The result is a quiet splitting internet: one version designed to delight human users, and another engineered to feed crawlers only specific content with the aim of protecting the value that’s still worth a click.