AI has borked web traffic - how are you handling it?
-
I like tech but Ai is really spooky.
Having said that, Ai is making a mess of web traffic among all the other things.
How are people handling the massive surges in users that are apparently Ai crawler bots and whatever else, acting like DDoS attacks without being intentionally malicious (assumption), the whole thing is a real headache.
There are many facets to this topic, one also being the proliferation of Ai into everything, and Serach being an obvious one, where the top result is now an AI readout, that is and will kill clicks down the SERP page and changes user behaviours, causing revenue loss already I am sure.
Removing the need to think. Making knowledge access even more frictionless to the point that broad scale cognitive atrophy occurs or is the need result, potentially further concentrating knowledge into smaller more powerful control groups (due to the current horsepower required). It's eating our energy supply too. Essentially we are at the BORG stage of the game.
As all should be rather familiar with now, it goes well beyond web traffic, but web traffic is the entry point here because this is where NodeBB's bread and butter is.
I flagged this in multiple places, years back at this point, with multiple warnings, adoption was done in haste, reaction was negative (as usual) and there was no due regard to Pandora's tsunami peopel thought they coudl surf, but the adoption of the Ai into anything and everything is what has allowed the Ai bot swarms to proliferate. it's a vicious cycle and maybe worse.
I could see and rationalise years ago that Ai itself is a parasitic technology and works in net deficit, it functions at a permanent net-negative (I understand many tech do not see it like this, feel free to posit your rebuttle). Why this is not glaringly obvious, is another point, but a fundamental one at that. There are other fundamental points. This is one that get's less airtime afaict.
Well have at it if you think you can add value to help the whole, server tips, what do you do to mitigate the loads, protect you content form the LLM plunder bots, etc. etc. - try stay on point in web traffic terms, as hard as it it due to the vastness of the consequences and implications before us, and Merry Christmas too while I'm here too!
Having found this topic
https://community.nodebb.org/topic/19021/any-protection-against-ai-crawlers-and-ai-learning-bots/
Thanks to user @shaknunic for introducing Anubis as one potential mitigation-solution
More info here:
>In an era where data is the new gold and AI is the new gatekeeper, safeguarding digital sovereignty has become more critical than ever. The rise of large-scale AI scraping has left countless small websites and independent developers struggling to keep their services online. Anubis was born from the urgent need to protect these digital voices from being drowned in a sea of automated traffic. As part of the broader mission of the AI & Data Foundation , this tool is not just a utility, it's a statement that the web belongs to everyone, not just those with the most aggressive crawlers.
>
>https://medevel.com/anubis-ai/ -
I like tech but Ai is really spooky.
Having said that, Ai is making a mess of web traffic among all the other things.
How are people handling the massive surges in users that are apparently Ai crawler bots and whatever else, acting like DDoS attacks without being intentionally malicious (assumption), the whole thing is a real headache.
There are many facets to this topic, one also being the proliferation of Ai into everything, and Serach being an obvious one, where the top result is now an AI readout, that is and will kill clicks down the SERP page and changes user behaviours, causing revenue loss already I am sure.
Removing the need to think. Making knowledge access even more frictionless to the point that broad scale cognitive atrophy occurs or is the need result, potentially further concentrating knowledge into smaller more powerful control groups (due to the current horsepower required). It's eating our energy supply too. Essentially we are at the BORG stage of the game.
As all should be rather familiar with now, it goes well beyond web traffic, but web traffic is the entry point here because this is where NodeBB's bread and butter is.
I flagged this in multiple places, years back at this point, with multiple warnings, adoption was done in haste, reaction was negative (as usual) and there was no due regard to Pandora's tsunami peopel thought they coudl surf, but the adoption of the Ai into anything and everything is what has allowed the Ai bot swarms to proliferate. it's a vicious cycle and maybe worse.
I could see and rationalise years ago that Ai itself is a parasitic technology and works in net deficit, it functions at a permanent net-negative (I understand many tech do not see it like this, feel free to posit your rebuttle). Why this is not glaringly obvious, is another point, but a fundamental one at that. There are other fundamental points. This is one that get's less airtime afaict.
Well have at it if you think you can add value to help the whole, server tips, what do you do to mitigate the loads, protect you content form the LLM plunder bots, etc. etc. - try stay on point in web traffic terms, as hard as it it due to the vastness of the consequences and implications before us, and Merry Christmas too while I'm here too!
The same big sources of disruptive traffic hitting everyone
Surprise surprise, Singapore is a hub for AI investment for a good number of years.
-
I like tech but Ai is really spooky.
Having said that, Ai is making a mess of web traffic among all the other things.
How are people handling the massive surges in users that are apparently Ai crawler bots and whatever else, acting like DDoS attacks without being intentionally malicious (assumption), the whole thing is a real headache.
There are many facets to this topic, one also being the proliferation of Ai into everything, and Serach being an obvious one, where the top result is now an AI readout, that is and will kill clicks down the SERP page and changes user behaviours, causing revenue loss already I am sure.
Removing the need to think. Making knowledge access even more frictionless to the point that broad scale cognitive atrophy occurs or is the need result, potentially further concentrating knowledge into smaller more powerful control groups (due to the current horsepower required). It's eating our energy supply too. Essentially we are at the BORG stage of the game.
As all should be rather familiar with now, it goes well beyond web traffic, but web traffic is the entry point here because this is where NodeBB's bread and butter is.
I flagged this in multiple places, years back at this point, with multiple warnings, adoption was done in haste, reaction was negative (as usual) and there was no due regard to Pandora's tsunami peopel thought they coudl surf, but the adoption of the Ai into anything and everything is what has allowed the Ai bot swarms to proliferate. it's a vicious cycle and maybe worse.
I could see and rationalise years ago that Ai itself is a parasitic technology and works in net deficit, it functions at a permanent net-negative (I understand many tech do not see it like this, feel free to posit your rebuttle). Why this is not glaringly obvious, is another point, but a fundamental one at that. There are other fundamental points. This is one that get's less airtime afaict.
Well have at it if you think you can add value to help the whole, server tips, what do you do to mitigate the loads, protect you content form the LLM plunder bots, etc. etc. - try stay on point in web traffic terms, as hard as it it due to the vastness of the consequences and implications before us, and Merry Christmas too while I'm here too!
I sincerely bring this to your attention urgently and for all, this is I believe a very good summary (link below) of what the net has just faced and what may be to come. I' not to worried about the cyber war what if's, but keeping the service, up, how to protect your service and retain function and aches for said users.
It aligns with my own web traffic experience, and thus reason for this topic, while reading anecdotally across reddit (e.g. link in previous post) and other platforms of others very recent traffic trials the bottom being, everyone seems to have been affected by this since at least November:
https://restoringthemind.com/china-singapore-bot-surge-raises-global-cyber-alarms/
Could this current wave be only a preemptive stress test and survey before a monumental attack?
At this new current level / wave it is breaking the open internet affairs
-
I like tech but Ai is really spooky.
Having said that, Ai is making a mess of web traffic among all the other things.
How are people handling the massive surges in users that are apparently Ai crawler bots and whatever else, acting like DDoS attacks without being intentionally malicious (assumption), the whole thing is a real headache.
There are many facets to this topic, one also being the proliferation of Ai into everything, and Serach being an obvious one, where the top result is now an AI readout, that is and will kill clicks down the SERP page and changes user behaviours, causing revenue loss already I am sure.
Removing the need to think. Making knowledge access even more frictionless to the point that broad scale cognitive atrophy occurs or is the need result, potentially further concentrating knowledge into smaller more powerful control groups (due to the current horsepower required). It's eating our energy supply too. Essentially we are at the BORG stage of the game.
As all should be rather familiar with now, it goes well beyond web traffic, but web traffic is the entry point here because this is where NodeBB's bread and butter is.
I flagged this in multiple places, years back at this point, with multiple warnings, adoption was done in haste, reaction was negative (as usual) and there was no due regard to Pandora's tsunami peopel thought they coudl surf, but the adoption of the Ai into anything and everything is what has allowed the Ai bot swarms to proliferate. it's a vicious cycle and maybe worse.
I could see and rationalise years ago that Ai itself is a parasitic technology and works in net deficit, it functions at a permanent net-negative (I understand many tech do not see it like this, feel free to posit your rebuttle). Why this is not glaringly obvious, is another point, but a fundamental one at that. There are other fundamental points. This is one that get's less airtime afaict.
Well have at it if you think you can add value to help the whole, server tips, what do you do to mitigate the loads, protect you content form the LLM plunder bots, etc. etc. - try stay on point in web traffic terms, as hard as it it due to the vastness of the consequences and implications before us, and Merry Christmas too while I'm here too!
@Julian I had to wait 5/10 mins before being able to post - coincidental, having hosting/server issues connecting, nodeBB community under pressure? Maybe from the same waves??
Meanwhile, a very common issues all around, below is Nov 29th reddit discussion, but I can see this issue being raised earlier in the year, September, October and maybe even earlier.
https://www.reddit.com/r/SEO/comments/1p9irqq/how_did_you_deal_with_the_chinasingapore_bot_or/
-
@Julian I had to wait 5/10 mins before being able to post - coincidental, having hosting/server issues connecting, nodeBB community under pressure? Maybe from the same waves??
Meanwhile, a very common issues all around, below is Nov 29th reddit discussion, but I can see this issue being raised earlier in the year, September, October and maybe even earlier.
https://www.reddit.com/r/SEO/comments/1p9irqq/how_did_you_deal_with_the_chinasingapore_bot_or/
@omega setting up Anubis is probably going to be the only path forward (especially if you don't want to use CF built in tooling.)
You want to let search crawlers through but stop AI crawlers. It's a tough game of cat and mouse.
-
@Julian Yea Anubis does look good but right now free CF tools and rules plucked in a need solution asap when this got going earlier in the year.
Here is the bad bot report for 2025 from Imperva it make for some interesting reading!
Like net traffic has tipped over the 50% mark for bot traffic first time in a decade.https://cpl.thalesgroup.com/sites/default/files/content/campaigns/badbot/2025-Bad-Bot-Report.pdf
-
This is a good run down of the wave the net is up against from a WP site perspective but applies to all.
I has already discovered that interactive challenge had a similar to blocking effect on the traffic using CF. It's taken a few re-gigs and tweaks to make the whole thing manageable since this became problematic in early November.
https://martech.zone/block-china-and-singapore-bot-traffic-using-cloudflare/
-
CDNs like Cloudflare seem to mitigate things to some degree. I self host a little instance of Mediawiki. Before putting it behind Cloudflare I was getting a firehose of requests, now it's just Cloudflare's caching thingy doing it's thing.
I'm not sure it does anything about AI crawlers other than taking the load off the end server and on to the CDN, so it's probably not stopping the clankers from stealing your data.
Using Cloudflare has its own host of issues though, namely concentrating a bunch of stuff behind a single point of failure as was seen a few weeks ago.
-
CDNs like Cloudflare seem to mitigate things to some degree. I self host a little instance of Mediawiki. Before putting it behind Cloudflare I was getting a firehose of requests, now it's just Cloudflare's caching thingy doing it's thing.
I'm not sure it does anything about AI crawlers other than taking the load off the end server and on to the CDN, so it's probably not stopping the clankers from stealing your data.
Using Cloudflare has its own host of issues though, namely concentrating a bunch of stuff behind a single point of failure as was seen a few weeks ago.
@anchorite Cloufflare now has a specific Ai feature with allow/block toggles for the various bots on the free tier. I'm not sure if you get more features in the paid tiers with that but you can write a lot more security rules.
-
@anchorite Cloufflare now has a specific Ai feature with allow/block toggles for the various bots on the free tier. I'm not sure if you get more features in the paid tiers with that but you can write a lot more security rules.
@omega nice! I shall give it a try... Wonder how well it works with federation
