Snapshot of the load ap.space sees from AI crawlers

[julian@activitypub.space]

Can you guess when I turned Anubis back on?

• Grey line (left-hand; y-axis) tracks page views
• Blue line (right-hand; y-axis) tracks unique users

You can even see the spike in traffic that brought down the site hard enough that I got my butt in gear to tune Anubis and turn it back on.

Based on the numbers here, there is a thirteen-fold decrease in activity (or a ~92% drop in traffic), all identified by Anubis as bots and blocked.

Selective adjustments were made to the nginx config and anubis bot policy to allow certain bits of traffic through (for unimpeded federation, etc.), but otherwise the site is now quite stable, on a small potato server.

Default bot policy does let search engine crawlers though (I think), so that is win-win.

Thank you so much @[email protected], my next stop is your GitHub Sponsors page.