Hi Tom First of all, thanks for trying to help!
Here are some details, feel free to skip them. There are several problems with your idea: 1. Gitlab is not very cacheable. Requests e.g. go to /random-user-fork/ghc/commits/random-commit-hash. And then you have exponentially many diffs between those commits. This is just an example. Even if we could enumerate all of them, I don’t think any cache would like to keep them up for us. Additionally, most routes are not static. 2. we want users to be able to use the instance without needing to log in. That includes issues, MRs, etc. 3. it is a large amount of work that doesn’t really fix any issues. The actual issue is - we can’t reliably distinguish users and spammers - we do have a lot of non-GHC developer users - Gitlab is not as performant as we’d like (we’re running this on an VERY beefy machine) There are a few things in reach for us though: - re-enabling Anubis for unauthenticated usage (this was disabled for good reasons though) - tarpitting (I think this actually works well but people had concerns about whether this would be too „mean“) - moving some other services off the machine. The rate limiting route is exhausted, the machine performance route is also exhausted. Adding yet more RAM to the machine seems like it’s just a waste of resources at this point. Every 2GB ram gives us one more puma worker and we’re already running 36 of them. The machine also has an absurd amount of ram either way, certainly enough to serve the ~200 actual people who are at worst using the instance at the same time. Best Magnus > On 23. Apr 2026, at 13:58, Tom Ellis via ghc-devs <[email protected]> > wrote: > > I'm not sure if this idea is any good, but: would it be possible and > desirable to have "public access" to the GHC GitLab go to a mirror, > and have authentication required on all access to the "live" instance > that actually serves developers? > > (This would require the live service to be behind a different domain > name, and would be a little bit like hand-rolling a special purpose > CDN.) > > Tom > >> On Thu, Apr 23, 2026 at 01:48:32PM +0200, Magnus Viernickel via ghc-devs >> wrote: >> since 12:00 we're getting another wave of spam. It's really hard to do >> something useful here, I will try more and more things but they will degrade >> UX for non-logged in users more and more. >> >> I'm prioritising UX for logged in users though. >> >> I'm unsure what to do to actually "solve" this issue. We're not seeing high >> traffic from singular IPs, we can't actually require log in for everything >> that people would like to do on the gitlab and gitlab itself is buggy in >> that it doesn't degrade well under load (now that I have fixed most of the >> other performance bottlenecks). >> >> I agree it's really frustrating and it's eating away at my personal time >> budget of working on actual GHC issues. >> >>> On 4/23/26 12:45, Simon Peyton Jones via ghc-devs wrote: >>> I'm still getting 20-second load times for individual pages (e.g. >>> tickets) on gitlab.haskell.org <http://gitlab.haskell.org>. >>> >>> Sometimes they errror out with "something went wrong, try again". >>> >>> Are we stuck with what to do? It's a bit frustrating. >>> >>> Simon >>> >>> >>> On Mon, 20 Apr 2026 at 14:15, Andreas Klebinger via ghc-devs >>> <[email protected]> wrote: >>> >>> Hello Devs, >>> >>> just a short update and some background about Gitlab issues today. >>> >>> Over the week and in particular today the number of requests to our >>> GitLab instance >>> has grown steadily, it now receiving way more than doubled the >>> number of >>> requests it >>> received last week. >>> >>> We have tried to solve this by adjusting some GitLab/server >>> settings to >>> deal with this >>> (rate limits, checking for spammy IPs, the works) but with little >>> success so far since it >>> seems that the requests are pretty distributed across their origins. >>> >>> Given it's not garbage requests but for valid endpoints we assume >>> it's >>> "just" crawlers >>> (likely driven by the AI hype) indexing our instance. But while this >>> isn't a DoS attack >>> in it's totality the effect similar and it's unclear when we will >>> make >>> progress on improving >>> the situation. >>> >>> Magnus is working towards resolving this issue and we will let you >>> know. >>> >>> Cheers >>> Andreas >>> >>> On 20/04/2026 10:52, Magnus Viernickel via ghc-devs wrote: >>>> Hello GHC devs, >>>> >>>> we're seeing a lot of 502s at the moment, we're getting twice >>> the rate >>>> of requests than normal, I am investigating what is going on and >>>> trying to fix this. > _______________________________________________ > ghc-devs mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ghc-devs mailing list -- [email protected] To unsubscribe send an email to [email protected]
