Hello GHC devs,
The gitlab should be fine again, it looks like the spammers have left on
their own.
Here is my analysis of the problem. If you have an idea on what to do or
are against my proposed "solution", please say so.
1. We had a much higher load than usual, leading to congestion on the
workers.
2. No matter what, we could not have added new workers, since either
they would get reaped via gitlabs normal worker reaping or they would be
OOMkilled when the entire application exceeded available RAM
3. Gitlab's own reverse proxy (yes, we have two) talks to the web
workers via a unix domain socket.
4. When we hit a certain load, the workers could not take requests from
the queue quickly enough
5. instead of stopping, the reverse proxy kept hammering the the queue
6. When the queue didn't fill up, you would just get a lot of client
cancels after a couple seconds
7. When the queue *did* fill up, the gitlab would start throwing
"service unavailable" errors *and immediately return with a 502*
8. Now a vicious cycle begins: the clients (the spammers) would
*immediately* resubmit their request instead of timing out, which kept
the queue congested. The only fix is to restart the gitlab, which of
course clears up the queue until it becomes congested again, from which
point on it kept being congested. (which is why I had a couple of
situations where I went "oh I fixed it" but then after half an hour and
a spike of traffic later, we were back to 502s)
The conclusion here is much sadder though:
1. I did an analysis of the origin and locked out a couple of bad /24
subnets but they only made up a tiny fraction of the traffic -- overall,
I could not identify "big bad subnets"
2. delegated blocks came from random (as far as cloud providers are
random given their are only three of them, we could e.g. see AWS) cloud
providers. This is expected, as bad actors of course wouldn't have their
own infra which would be pretty easy to ban wholesale
3. This is why IP address based blocking (i.e. rate limiting) did not
work, which we're doing on the gitlab level (per IP) and on the reverse
proxy level (for /24s) (our nginx, not gitlab's own reverse proxy).
4. The only thing that would really help is more workers, which also
isn't a great fix. They could just hammer with more requests *and* the
workers are taking up an absurd amount of RAM (currently about 50GB,
every worker is 1-2 more GB, so convervatively one worker is 2 GB of ram
extra that we need)
5. Or to reintroduce anubis or some tarpit on the expensive routes.
Maybe this is one of the easiest fixes. We would of course only have
this for unauthenticated requests and on expensive routes. The worst
offender being graphql requests.
Best
Magnus
On 4/20/26 15:15, Andreas Klebinger via ghc-devs wrote:
Hello Devs,
just a short update and some background about Gitlab issues today.
Over the week and in particular today the number of requests to our
GitLab instance
has grown steadily, it now receiving way more than doubled the number
of requests it
received last week.
We have tried to solve this by adjusting some GitLab/server settings
to deal with this
(rate limits, checking for spammy IPs, the works) but with little
success so far since it
seems that the requests are pretty distributed across their origins.
Given it's not garbage requests but for valid endpoints we assume it's
"just" crawlers
(likely driven by the AI hype) indexing our instance. But while this
isn't a DoS attack
in it's totality the effect similar and it's unclear when we will make
progress on improving
the situation.
Magnus is working towards resolving this issue and we will let you know.
Cheers
Andreas
On 20/04/2026 10:52, Magnus Viernickel via ghc-devs wrote:
Hello GHC devs,
we're seeing a lot of 502s at the moment, we're getting twice the
rate of requests than normal, I am investigating what is going on and
trying to fix this.
Best
Magnus
_______________________________________________
ghc-devs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ghc-devs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ghc-devs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
