We have seen this also. It seems like AI-content harvesting. Unfortunately these robots that do not honour the robots.txt and are poor crawlers, so they not only hit the server from multiple IP addresses. They typically have an anonymous user-agent string. They also get stuck in an endless crawl of browse and search discovery end points. Our experience has been that these can be difficult to block as they IP ranges change are fairly dynamic. We have introduced some apache-level filters to block searches that we can identify as bot-created rather than human (they query string is often irregular). This helps quite a lot. We are also looking at more detailed inspection to throw up a confirmation page where the activity is a suspected bot.
If they are harvesting a content page, we generally let them go through on the general principle that it is better they populate their AI with "good" content from a curated repository. DSpace v8 is also quite a bit better at cached delivery and SEO. However when they hit the search end points they can easily overwhelm the server. Edmund On Thursday, April 24, 2025 at 3:29:16 AM UTC+10 Carolyn Sullivan wrote: > Hello all, > > We've been getting a ton of traffic from IP addresses with Alibaba Cloud > out of Hong Kong and Singapore that's been overwhelming our servers. Is > there any chance that legitimate users could be using Alibaba as an ISP, or > would any accesses for Alibaba be coming from programs running on the cloud > that we can unrepentantly block? > > Thanks, > Carolyn. > -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/dspace-tech/b3d72f5d-736b-4991-80b8-4b9ade9a47c3n%40googlegroups.com.
