We have seen this also.  It seems like AI-content harvesting.  
Unfortunately these robots that do not honour the robots.txt and are poor 
crawlers, so they not only hit the server from multiple IP addresses.  They 
typically have an anonymous user-agent string.  They also get stuck in an 
endless crawl of browse and search discovery end points.   Our experience 
has been that these can be difficult to block as they IP ranges change are 
fairly dynamic.  We have introduced some apache-level filters to block 
searches that we can identify as bot-created rather than human (they query 
string is often irregular).  This helps quite a lot.  We are also looking 
at more detailed inspection to throw up a confirmation page where the 
activity is a suspected bot. 

If they are harvesting a content page, we generally let them go through on 
the general principle that it is better they populate their AI with "good" 
content from a curated repository. 

DSpace v8 is also quite a bit better at cached delivery and SEO.  However 
when they hit the search end points they can easily overwhelm the server.

Edmund  
On Thursday, April 24, 2025 at 3:29:16 AM UTC+10 Carolyn Sullivan wrote:

> Hello all,
>
> We've been getting a ton of traffic from IP addresses with Alibaba Cloud 
> out of Hong Kong and Singapore that's been overwhelming our servers.  Is 
> there any chance that legitimate users could be using Alibaba as an ISP, or 
> would any accesses for Alibaba be coming from programs running on the cloud 
> that we can unrepentantly block?
>
> Thanks,
> Carolyn.
>

-- 
All messages to this mailing list should adhere to the Code of Conduct: 
https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
--- 
You received this message because you are subscribed to the Google Groups 
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion visit 
https://groups.google.com/d/msgid/dspace-tech/b3d72f5d-736b-4991-80b8-4b9ade9a47c3n%40googlegroups.com.

Reply via email to