Over the past year, traffic across our servers has climbed sharply. Not from people, though. The surge has come from AI bots.
Since early 2025, nearly all of our hosted sites have been receiving a steady increase in automated requests from AI crawlers. These are run by companies building large language models and search tools, gathering data to train and improve their systems. For content-rich, search-enabled sites like ours, that makes for a lot of tempting material to index.
More traffic sounds good on the surface. It means client content is being noticed and used. But there’s a catch: not all bots behave well.
Balancing openness with protection
The goal has been a careful balance: let legitimate AI crawlers (Googlebot, Bingbot, GPTBot and the like) access content, while stopping rogue or overly aggressive bots from overloading shared servers.
To do that, we’ve set a limit of 100 page requests per 5 minutes per IP address. That’s comfortably more than normal crawling needs. We tested this ourselves with DeepSeek and ChatGPT, and neither came close to the limit; both mimic human browsing patterns rather than bulk-downloading thousands of pages in one pass.
When a crawler does exceed normal behaviour, the system automatically returns a 429 (Too Many Requests) response, telling the bot to slow down. This is now a widely accepted standard, and it protects performance and stability for every site on a shared server.
What we’ve seen so far
Since rolling out this safeguard on our AWS servers, the results have been clear. No reputable crawler has been blocked, and Google hasn’t hit the rate limit once. What has been caught are the less desirable visitors: traffic from duckduck.go, mojeek.com, and even ebook-search-queen.com, all now safely throttled back.
Worth noting: while major players like Google say they follow industry best practice and respect robots.txt, real-world experience suggests that’s not always the case. Cloudflare’s write-up on the shift from Googlebot to GPTBot is a good overview of how this landscape is changing.
→ Cloudflare - From Googlebot to GPTBot: who’s crawling your site in 2025?
Why we can’t ‘whitelist’ every bot
We looked at building an allow list for specific AI bots. But AWS’s managed security infrastructure deliberately limits rule complexity to protect performance, and stacking up bot-specific exceptions would undermine the exact stability we’re trying to preserve.
Moving forward
Clients who genuinely need the rate limit lifted, say, for large-scale crawling of thousands of pages in minutes, can move to a dedicated server. That gives full control over the limits, at the cost of higher hosting.
For everyone else, the current setup strikes the right balance of accessibility, fairness and protection: visible to legitimate crawlers, fast and stable for real users.
The rise of AI-driven crawling is changing the web in real time. Our job is to make sure those changes don’t slow you down.