On Mon, Feb 20, 2012 at 05:57:05PM +0100, Roger S. wrote:
> I am facing regular and consequent DDoS, and I would like to know how
> the OpenBSD community deal with these. Hints and inputs welcome.
>
> The obvious first : my input pipes are not filled, there is plenty of
> bandwith available for my regular users. (...)
>
> Methodology is more or less always the same :
> - massive UDP flood : 2 Gbps / 150 Kpps -> dropped
> directly on the router, not a problem
> - moderate ICMP flood : 10 Mbps / 12 Kpps
> - moderate IP fragments flood : 380 Mbps / 57 Kpps
> - moderate TCP RST flood : 10 Mbps / 30 Kpps
> - massive TCP SYN flood : 640 Mbps / 2 Mpps -> yup, that hurts
>
> So, UDP never ever reaches my OpenBSD box. The SYN are made with a
> very vicious method : each used IP send exactly one SYN, but there are
> millions of them (traffic probably spoofed, but can not use uRPF as we
> have asymmetric traffic and routes). I tried to set limit states with
> 1M entries, and it was quickly filled (tried 5M but the box collapses
> way before that). So in the end, the state table collapses and no
> traffic can pass, even for regular users with already established
> connections.
>
> I ran some experiments in a lab trying to reproduce this, with a box
> roughly identical to what I have in production (but much weaker, of
> course). The box collapses at 600 Kpps SYN (100% interrupts), but
> handles everything very gently (less than 50% interrupts and no packet
> loss) if the first rule evaluated is block drop in quick from !
> <whitelisted_users>. So it seems that my bottleneck is PF here, not
> the hardware. A consequence of this saturation : both my main firewall
> and my backup claims MASTER ownership of the CARP (split brain
> syndrome). CARP works just fine when I add the block rule, though.
>
> Some configuration details :
> - OS : OpenBSD 5.0/amd64 box, using GENERIC.MP
> - CPU : Intel X3460 CPU (4 cores, 2.80GHz)
> - RAM : 4GB
> - NIC : 2x Intel 82576 (2 ports each)
>
> Each network card has the following setup : one port to the LAN, one
> port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is trunked using
> LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
> supported. My benchmarks did highlight two interesting things : amd64
> has better performance than i386 (roughly 5-10% less interrupts, with
> same rules and traffic), but the difference between GENERIC and
> GENERIC.MP is insignificant.
>
> My current idea is to hack a daemon to track established connections
> (extracting them ` la netstat), and inject my block rule in an anchor
> (` la relayd) when needed (watching some stats from pf, with its ioctl
> interface). Pros: regular users the firewall saw before the attack can
> still use the service. Cons: no new users are allowed until the
> removal of the rule, obviously. Better than nothing, but I welcome any
> other hints :)
>
> One other solution may be to add boxes. I tried a carpnodes cluster,
> but at 600 Kpps I got a "split brain" with both nodes claiming MASTER
> for each carpnode. Maybe if I configure ALTQ it could help this ? As I
> have more boxes, I could deal with the performance impact of ALTQ.
>
> I am willing to test any patch/suggestion you may have, of course.
> Even just hints about kernel code, as I am currently messing with PF
> code myself. I did compile a profiled kernel, I must now check the
> results but that will be another story.
Just the most obvious idea, since you mention that this sort-of-works if
you put "block drop in quick from !<whitelisted_users>": does it handle
this load if you turn off pf, or only include one or two trivial rules?
It certainly suggests that you may be well-served by optimizing your
pf.conf... (also, you've probably found the "synproxy" directive? If
not, try that too.)
Also, state tracking is apparently faster than stateless pf for normal
firewalls. I'd double-check if this is still true in your case, though;
if nothing else, stateless pf makes a CARP'ed setup easier.
I'm pretty sure you can muck with the rules without dropping existing
connections. (pf essentially does "does this packet match a known state?
If not, look at pf.conf".) This is almost certainly easier than your
proposed daemon.
A final, rather hackish, idea that probably does need a bit of
programming: greylisting for SYNs. Legitimate users will send you a
second SYN, so you could do something like (this has not even been
syntax-checked!)
block drop log in quick from !<syn_seen> no state flags S/SA
and then add every logged IP to syn_seen. Obviously, this will slow down
access to the service for legitimate users, which may or may not be
acceptable.
Joachim
--
PotD: www/squid,ntlm - WWW and FTP proxy cache and accelerator
http://www.joachimschipper.nl/
