Shaknunic here's the catch-22... If you want to stop AI crawlers, you can stop the good ones using robots.txt
https://github.com/ai-robots-txt/ai.robots.txt
If you want to stop the bad ones (which you probably do)... you can't
They masquerade as legitimate users, and they have more funding behind them than you have time. Every time you try to block them someone on the other end will adapt.
Unfortunately right now the only reliable solution is a hard challenge from CloudFlare for nearly all requests.
