If your website logs and analytics have started looking a little robotic lately, you’re not alone. The internet is changing fast. What was once a space dominated by human visitors is now increasingly trafficked by bots, especially AI-powered crawlers scraping the web for data to train the next generation of LLMs (large language models).
At a recent Akamai webinar, the numbers were stark: bots now account for nearly half of all internet traffic, and that share is growing. For website owners, this isn’t just a curiosity, it’s a shift that affects performance, security and even how we think about published content.
Good bots vs bad bots: know the difference
Not all bots are created equal.
Good bots, like GoogleBot or OpenAI’s GPTBot, identify themselves clearly, respect robots.txt, and may bring value. They crawl responsibly and often provide opt-out or throttling mechanisms.
Bad bots are the troublemakers. They hide behind fake user-agent strings, ignore crawl directives, scrape content without permission, and can slow your site to a crawl or worse, siphon off sensitive data. These bots offer no value, only risk. We’ve covered them in more depth in a previous article, “When the bots came knocking: how we’re managing the AI crawl surge”.
The silent shift: more bots, fewer people
As AI bot activity surges, the proportion of people browsing your website is shrinking. Not necessarily in absolute numbers, but as a share of overall traffic. That has real consequences:
- Performance hits - aggressive crawlers can degrade the experience for actual visitors.
- Muddled analytics - telling human behaviour from bot behaviour apart is harder than ever.
- Rising resource costs - servers work overtime responding to non-human requests.
Rethinking content for bots and people
This isn’t just a traffic-management issue, it’s a content strategy imperative. To stay visible and valuable in the age of AI search, we need to publish with both audiences in mind:
- Structure for clarity - use clear headings (H1, H2, H3), semantic HTML and structured data (Schema.org) so bots can understand and categorise your content accurately.
- Write for intent - answer questions directly and comprehensively. AI tools favour content that clearly addresses user queries, not keyword-stuffed pages.
- Embrace E-E-A-T - demonstrate Experience, Expertise, Authoritativeness and Trustworthiness. This matters more than ever as AI models learn to prioritise credible sources.
- Refresh regularly - bots and people both prioritise fresh, updated content. Set a schedule for reviewing and revitalising key pages, and stick to it.
Taking back control
So what can you actually do? Start by knowing who’s knocking. Regularly review your server logs, or use a tool like Cloudflare’s Verified Bots library to tell friendly crawlers from hostile ones.
From there:
- Update
robots.txtto block unwanted AI training crawlers (like Google-Extended or GPTBot). - Use bot management modules if you’re on a CMS like Drupal.
- Consider a Web Application Firewall (WAF) with advanced bot detection to filter traffic at the edge.
The goal isn’t to block every bot, many are essential, but to manage them intelligently, so your site stays fast, secure and accessible for the people who matter most: your actual users.
Closing thoughts
The bot surge is real. People no longer make up the majority of visitors to most sites. But they’re still the audience that matters most. By understanding bot traffic, structuring content for clarity, and taking proactive steps to manage crawlers, you protect your site and future-proof it for an era when AI doesn’t just visit your content, it learns from it, shares it, and decides who gets to see it.
Stay curious, stay structured, and keep building for people.