Unknown actors are running mass vulnerability scans against websites while masquerading as legitimate AI crawlers like Anthropic's ClaudeBot. The attacks exploit a fundamental weakness in how the web identifies automated traffic: the User-Agent string is trivially spoofed, and most sites have no way to verify whether the bot claiming to be ClaudeBot is actually operated by Anthropic.
Fresh data from Known Agents, which tracks bot activity across more than 5,000 websites, puts numbers to the problem. Bots now account for 35% of all web traffic on monitored sites. Of that bot traffic, 29% is AI-related, an 11-percentage-point jump compared to the previous 90 days. The share of traffic that websites can confidently attribute to legitimate purposes is shrinking.
What the traffic breakdown reveals
Search engine crawlers still dominate, with Googlebot and Bingbot together accounting for about 16% of all bot activity. But the rest of the top ten is a mix of SEO tools, AI assistants, and data scrapers. ChatGPT-User, the crawler OpenAI uses when a user asks ChatGPT to fetch live web content, represents 3.3% of bot traffic. ClaudeBot, which Anthropic uses to train its models, accounts for 3.2%.
Those percentages matter because they represent real crawling volume. AhrefsBot, the crawler behind Ahrefs, hits 6.3%. Semrush's SemrushBot is at 2.9%. Meta runs two separate scrapers, facebookexternalhit (2.6%) and meta-externalagent (2.3%). Amazon operates both Amazonbot (2.3%) and Amzn-SearchBot (2.2%).
Disclosure
Some links in this post are affiliate links — Logicity earns a commission if you sign up, at no extra cost to you. We only link products we have used or actively recommend.
The Known Agents taxonomy splits these into 16 categories. "AI Data Scraper" means the bot downloads content to train models. "AI Search Crawler" indexes content for AI-powered search results. "AI Assistant" fetches pages in response to user prompts. "Undocumented AI Agent" is the catch-all for crawlers that do not disclose their purpose. That last category is small, just 0.2% of traffic, but it is growing.
Why robots.txt cannot solve this
The 98.5% compliance figure sounds reassuring. Almost all bots honor robots.txt. But that number measures bots that identify themselves honestly. A security scanner pretending to be ClaudeBot can ignore robots.txt entirely, and the server has no way to know.
Robots.txt was designed for a cooperative web. You publish a file saying "do not crawl /admin," and well-behaved bots respect it. The entire system assumes good faith. There is no authentication, no signature, no way to verify that the request claiming to come from ClaudeBot actually originates from Anthropic's infrastructure.
Anthropic publishes the IP ranges ClaudeBot uses. So does OpenAI for GPTBot. A site operator can, in theory, cross-reference incoming requests against those ranges and reject anything that claims to be an AI crawler but comes from an unexpected IP. In practice, almost nobody does this. It requires ongoing maintenance, because IP ranges change, and most web frameworks do not make it easy.
The spoofing attack surface
An attacker running vulnerability scans has good reasons to impersonate AI crawlers. Many sites have added AI bots to their allowlists, eager to appear in AI-generated answers or to avoid blocking legitimate model training. A scanner claiming to be ClaudeBot might slip past rate limits or WAF rules that would block an unknown User-Agent.
The traffic pattern is distinctive. Legitimate AI crawlers request pages that contain useful training data: documentation, blog posts, product pages. A vulnerability scanner posing as ClaudeBot will probe for exposed admin panels, debug endpoints, and files like .env or .git/config. If your logs show ClaudeBot requesting /wp-admin/install.php or /.aws/credentials, it is not Anthropic.
The Known Agents data does not break out spoofed traffic specifically. The 3.2% figure for ClaudeBot includes both legitimate and fake requests, because the tracking relies on User-Agent strings. The true number of scans hiding behind AI crawler identities is unknown, but security researchers have observed the pattern in the wild.
What site operators can do
IP verification is the only reliable defense. Anthropic, OpenAI, Google, and other major bot operators publish their IP ranges. A firewall rule or edge function that checks incoming requests against those ranges can reject spoofed traffic.
Cloudflare offers Bot Management that goes beyond User-Agent checks. It uses behavioral signals, machine learning, and request fingerprinting to distinguish real crawlers from imposters. Fastly and AWS WAF provide similar capabilities, though they require more manual configuration.
For sites without enterprise WAF budgets, the minimum viable defense is logging. Record the full User-Agent, the source IP, and the requested path. Review logs for AI crawler User-Agents requesting paths no legitimate crawler would care about. If you see ClaudeBot hitting /.git/config, block that IP.
Logicity's Take
The "agentic web" is here, and it is already being weaponized. AI companies have created valuable identities, bot names that many sites explicitly whitelist. Attackers are exploiting that trust. Founders should assume that any traffic claiming to be an AI crawler might not be, and build verification into their security posture now. The alternative is waiting until you learn from an incident.
The broader shift in web traffic
The 35% bot traffic figure is down 1% from the previous quarter, but the composition is changing fast. AI-related bots grew 11 points. AI chat referrals, the traffic that arrives when a user clicks a link in ChatGPT or Claude, sit at just 0.1% of human visits. The gap between how much AI bots take from the web versus how much traffic they send back is stark.
For startups, this raises strategic questions. Should you block AI training crawlers to protect your content, or allow them to appear in AI-generated answers? The tradeoff was already complex. Adding "you cannot trust that ClaudeBot is actually ClaudeBot" makes it worse.
The spoofing problem will get worse before it gets better. As more sites whitelist AI crawlers, the incentive to impersonate them grows. The web's identity layer, built on cooperative assumptions that made sense in 1994, is not equipped for an adversarial AI ecosystem. Something will have to change, but it is not clear what, or who will build it.
Need Help Implementing This?
If you need help setting up IP verification for AI crawlers or reviewing your bot traffic patterns, reach out to Logicity's consulting team.
Source: Hacker News: Best
Huma Shazia
Senior AI & Tech Writer
Produced with AI assistance and reviewed by the Logicity editorial team. Learn more in our Editorial Policy.
