Cloudflare CEO Matthew Prince predicted that bots would outnumber human visitors by 2027. He was wrong. It happened two years early.
According to data reported by NBC News, 57.2% of HTTP requests to HTML content are now generated by bots, compared to 42.8% from humans. Before the generative AI era, bot traffic sat at around 20% of all web activity. The crossover that was supposed to arrive next year has already passed, and the implications for eCommerce merchants are immediate and practical.
"Welp, that happened faster than I predicted," Prince wrote on X. "Agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet's history."
Why this matters for your store
More than half of the traffic hitting your website right now is not human. It is automated scripts, AI crawlers, scrapers, price comparison bots, inventory checkers, and, increasingly, AI agents researching on behalf of users. For eCommerce merchants, this is not an abstract infrastructure concern. It directly affects your costs, your data quality, your competitive position, and your ability to understand your own customers.
Server costs. Every bot request consumes bandwidth, compute cycles, and CDN resources. If 57% of your traffic is automated, you are potentially paying nearly double what you should for infrastructure. For merchants on usage-based hosting or CDN plans, this translates directly to higher monthly bills.
Analytics distortion. Your bounce rate, session duration, and conversion rate metrics are all calculated against total traffic. If a significant portion of your sessions are bots that your analytics platform fails to filter, your actual human conversion rate is higher than reported and your marketing ROI calculations are based on inflated denominators. This leads to systematically undervaluing your marketing spend.
Inventory and pricing exposure. Competitor bots that scrape your product catalogue, pricing, and inventory levels operate at scale. If you are running dynamic pricing or limited inventory promotions, these bots can expose your strategy to competitors in near real-time.
The AI crawler dimension
What makes this moment different from previous bot traffic discussions is the composition. A growing portion of these automated requests comes from AI companies crawling the web to train and operate their models. OpenAI's GPTBot, Google's AI crawlers, Anthropic's ClaudeBot, and dozens of smaller operators are systematically indexing eCommerce content.
This creates a paradox for merchants. You want your products to appear in AI-generated recommendations and search results, particularly as users increasingly outsource purchase decisions to AI agents. But you also need to control how your content is consumed, at what rate, and by whom.
As Prince explained in an interview with TechCrunch at SXSW earlier this year, a human shopping for a digital camera might visit five websites, while an AI agent performing the same task might visit 5,000. That volume of automated browsing is real traffic and real server load that every merchant needs to account for.
What you should actually do
Audit your bot traffic
Start by understanding the scale of the problem on your specific site. Most CDN providers, including Cloudflare, Fastly, and Akamai, offer bot traffic reports. If you are on Cloudflare, the Bot Management dashboard will show you what percentage of your traffic is automated and which bots are responsible.
On Magento or Adobe Commerce, check your server access logs for user agent strings. Common AI crawlers identify themselves (GPTBot, ClaudeBot, Bytespider), but many scrapers deliberately disguise themselves as regular browsers.
Implement tiered bot management
Not all bots are equal. Search engine crawlers such as Googlebot and Bingbot are essential for your discoverability. AI crawlers may or may not serve your interests. Price scrapers and inventory checkers are almost certainly adversarial.
Use your CDN or web application firewall to implement tiered policies. Allow legitimate search engine crawlers. Rate-limit AI crawlers to prevent resource abuse while maintaining discoverability. Challenge unidentified or suspicious automated traffic. Block known malicious scrapers.
Control AI crawler access via robots.txt
Use robots.txt to manage which AI crawlers can access your content and which pages they can index. This is now a strategic decision, not just a technical one. Do you want your product descriptions to appear in AI-generated shopping recommendations? If so, allow AI crawlers but rate-limit them. If you are concerned about content being used for competitive training, restrict access.
Recalibrate your analytics
If you have not already done so, enable bot filtering in Google Analytics and verify that your analytics platform is correctly excluding known bot traffic. Then recalculate your key metrics, including conversion rate, cost per acquisition, and return on ad spend, against filtered traffic only. You may find your actual performance is significantly better than your dashboards suggest.
Protect your pricing and inventory data
If you run dynamic pricing, consider implementing rate limiting on product API endpoints and category pages. Ensure your pricing data is not easily extractable through structured data markup without corresponding access controls. This is especially relevant for B2B merchants who offer negotiated pricing that should not be publicly visible.
The bigger picture
The 57% threshold is not the end state. It is a milestone on a clear trajectory. As AI agents become more capable and more widely deployed, the proportion of automated traffic will continue to grow. The merchants who treat this as a strategic challenge rather than a technical nuisance will be better positioned to manage costs, protect their competitive advantages, and ensure they remain visible in the AI-mediated discovery channels that increasingly influence purchase decisions.
The web is becoming a place where machines talk to machines on behalf of humans. Your store needs to be ready for both audiences.
About On Tap
On Tap is a growth-focused eCommerce consultancy helping mid-market and enterprise merchants navigate the technical and strategic challenges of modern commerce. From infrastructure optimisation and bot management strategy to analytics configuration and platform performance, On Tap works across the full eCommerce stack to help merchants protect margins and build for the environment that actually exists.
If bot traffic is affecting your store's performance or your analytics confidence, get in touch.
