Dear GHC Devs,
I have some interesting information to share with you.
You might have noticed that our gitlab was a bit more swamped the recent
days again and today I investigated and discovered that Anubis was
getting bypassed by requests with a new user agent that I haven't seen
before, "Lightpanda/1.0".
Normally, anubis checks for user agents that are known bot user agents
and blocks them; Lightpanda was left unblocked because Anubis only
filtered for LightPanda (capital p).
This is normally not a problem because most crawlers try to use user
agents that look like browsers; if you're using a browser user agent,
you're getting a higher weight and you get presented as challenge; but
you're also much harder to sift out because you look benign.
While the amount of requests we were getting with that user agent had
reached absurd dimensions, this was also a blessing. First of all, it
was easy to ban it and reduce the gitlab load drastically.
Second, now we could run an analysis on who is using these malicious
clients. Xe, the creator of Anubis kindly provided me with a tool that
qualitatively (based on unique IP addresses, disregarding hits per IP)
puts IP addresses in buckets based on several known factors, e.g.
location or the ASN (which entity the IP address block is delegated to).
The results can be seen here: https://gitlab.haskell.org/-/snippets/6132
A few noteworthy things:
- 1.4 Million distinct IPv4 addresses have been used with this bot user
agent
- the IP addresses stem from 225 different countries. I think we have ~
30 more distinct ISO country codes.
- we have some known faces! like spacex which got flagged 100% time on
their 21000 distinct IP addresses they're using to DDoS us
- we also have quite a lot of domestic use ASNs, which indicates that
botnets are involved, too!
- Xe noted, that we can proudly say this counts as a DDoS
Mind that this does not include all the spammers that are actually
trying to hide the fact that they're spammers, which still exist. Also
mind that this was just the data from ~1 day of logs.
I hope this gives a nice sense of what we're dealing with and maybe it
also helps with your frustration to see my frustration and the fact that
we're sometimes getting little wins. Like today.
Best
Magnus
_______________________________________________
ghc-devs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
