Re: [Evergreen-general] Dealing with significant traffic increase caused by AI bots

Thank you for sharing the link to the Dark Visitors website - it looks very useful, indeed! Linda On 4/19/24 20:21, Lolis, John via Evergreen-general wrote: There's been quite a conversation on the CODE4LIB listserv about this lately... Scott Prater

Re: [Evergreen-general] Dealing with significant traffic increase caused by AI bots

There's been quite a conversation on the CODE4LIB listserv about this lately... Scott Prater <007dd2c67ad2-dmarc-requ...@lists.clir.org> Thu, 11 Apr, 10:43 (8 days ago) to CODE4LIB We've also been seeing some traffic from inconsiderate AI bots. One of my colleagues came across this site,

Re: [Evergreen-general] Dealing with significant traffic increase caused by AI bots

Thank you very much, Jane! We will certainly give fail2ban a try, though - as we use Apache - some implementation details will probably be a bit different :-). Linda On 4/19/24 13:05, Jane Sandberg wrote: Hi Linda, It's not for Evergreen, but my colleague recently blocked claudebot using

Re: [Evergreen-general] Dealing with significant traffic increase caused by AI bots

Hi Linda, It's not for Evergreen, but my colleague recently blocked claudebot using fail2ban on our load balancer . Essentially, fail2ban is configured to watch Nginx's access log, and if more than 10

[Evergreen-general] Dealing with significant traffic increase caused by AI bots

Dear all, Have any of you encountered an extensive crawling by Bytespider and Bytedance (see e.g., https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/), Claudebot or other AI bots? If so, do you have any secret recipe how to disable the crawler from