Re: D/DoS mitigation hardware/software needed.
On Mon, 11 Jan 2010, jul wrote: Known leader of the clean-pipe solution is Prolexic http://www.prolexic.com/ Akamai and Verisign also tries to go on this market http://www.akamai.com/security (through CDN) http://www.verisign.com/internet-defense-network/index.html Indeed, these 3 also ended up on my shortlist after much research. Each has areas they are weaker in than their competitors but they are all worthy. -Hank
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Hank Nussbacher [mailto:h...@efes.iucc.ac.il] Sent: Monday, January 11, 2010 4:40 AM To: jul Cc: NANOG Subject: Re: D/DoS mitigation hardware/software needed. On Mon, 11 Jan 2010, jul wrote: Known leader of the clean-pipe solution is Prolexic http://www.prolexic.com/ Akamai and Verisign also tries to go on this market http://www.akamai.com/security (through CDN) http://www.verisign.com/internet-defense-network/index.html Indeed, these 3 also ended up on my shortlist after much research. Each has areas they are weaker in than their competitors but they are all worthy. If you're connected to Level 3 in any capacity, they have a reseller agreement with Prolexic and can offer REALLY aggressive pricing. I really liked their offering. If anyone is interested, I did pretty exhaustive research into the Service Provider marketplace last summer (before Verisign came out with their VIDN). I've got some slides which outline the costs, mitigation capacity, etc. of many different providers. The provider option isn't always the cheapest when compared to DIY factored in over a 3-5 year lifespan. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 11, 2010 2:05 AM On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. Don't forget to factor in OpEx. This can often tilt the scales in favor of one vs. the other. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
I thought I had mentioned outsourcing earlier, but I don't see it in the thread... The two mechanisms I've seen for outsources D/DoS are DNS manipulation, or essentially remote BGP peering with an tunnel back to the local presence. Even if we are purely hosting, DNS manipulation doesn't do anything for attacks against an IP. For remote BGP peering/tunneling, you are are adding additional complexity and moving control outside your network. As a service-provider/data-center, it seems like outsourcing would be either ineffective and/or removes the big red button in case of trouble. Am I missing something, overly paranoid, or are there other mechanisms for outsourced protection? Rick On Mon, Jan 11, 2010 at 6:33 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 11, 2010 2:05 AM On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. Don't forget to factor in OpEx. This can often tilt the scales in favor of one vs. the other. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, January 11, 2010 10:39 AM To: NANOG Subject: Re: D/DoS mitigation hardware/software needed. As a service-provider/data-center, it seems like outsourcing would be either ineffective and/or removes the big red button in case of trouble. Am I missing something, overly paranoid, or are there other mechanisms for outsourced protection? In fact, quite the opposite. Those providers who do offer DDoS mitigation services usually allow the customer to trigger the redirect in a manner similar to RTBHs by substituting the blackhole community for some type of mitigation community. This causes the Provider's edge router (or Route Server) to advertise the affected route within the Service Provider's network with a next-hop of the scrubbers. There are some providers who do auto-mitigation on behalf of the customer, but IMO this approach is asking for trouble. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 11, 2010 at 9:33 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 11, 2010 2:05 AM On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. Don't forget to factor in OpEx. This can often tilt the scales in favor of one vs. the other. sure, but just capex alone the 'make vzb do this' option wins. (I think at least, I'm not a math guy though...)
RE: D/DoS mitigation hardware/software needed.
Precisely - I was saying that in order to add more point to your argument. I wasn't disagreeing with you :) Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D -Original Message- From: christopher.mor...@gmail.com [mailto:christopher.mor...@gmail.com] On Behalf Of Christopher Morrow Sent: Monday, January 11, 2010 12:49 PM To: Stefan Fouant Cc: jul; NANOG Subject: Re: D/DoS mitigation hardware/software needed. On Mon, Jan 11, 2010 at 9:33 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 11, 2010 2:05 AM On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. Don't forget to factor in OpEx. This can often tilt the scales in favor of one vs. the other. sure, but just capex alone the 'make vzb do this' option wins. (I think at least, I'm not a math guy though...)
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 11, 2010 at 1:12 PM, Stefan Fouant sfou...@shortestpathfirst.net wrote: Precisely - I was saying that in order to add more point to your argument. I wasn't disagreeing with you :) i need more coffee :( thanks! -Chris
Re: D/DoS mitigation hardware/software needed.
Right. Some providers allow you to BGP community trigger RTBH. There was a separate mention of D/DoS-mitigation-providers using DNS and BGP tunneling. Rick On Mon, Jan 11, 2010 at 8:14 AM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, January 11, 2010 10:39 AM To: NANOG Subject: Re: D/DoS mitigation hardware/software needed. As a service-provider/data-center, it seems like outsourcing would be either ineffective and/or removes the big red button in case of trouble. Am I missing something, overly paranoid, or are there other mechanisms for outsourced protection? In fact, quite the opposite. Those providers who do offer DDoS mitigation services usually allow the customer to trigger the redirect in a manner similar to RTBHs by substituting the blackhole community for some type of mitigation community. This causes the Provider's edge router (or Route Server) to advertise the affected route within the Service Provider's network with a next-hop of the scrubbers. There are some providers who do auto-mitigation on behalf of the customer, but IMO this approach is asking for trouble. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
Stefan Fouant wrote on 11/01/10 14:45: If anyone is interested, I did pretty exhaustive research into the Service Provider marketplace last summer (before Verisign came out with their VIDN). I've got some slides which outline the costs, mitigation capacity, etc. of many different providers. The provider option isn't always the cheapest when compared to DIY factored in over a 3-5 year lifespan. If you can share, I think everyone would be interested. I am :)
RE: D/DoS mitigation hardware/software needed.
Ummm... there is some proprietary information I would have to remove first. Will NANOG accept a message to the forum with an attachment? If not I can put it up on my site. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D -Original Message- From: jul [mailto:jul_...@yahoo.fr] Sent: Monday, January 11, 2010 4:58 PM To: Stefan Fouant Cc: 'Hank Nussbacher'; 'NANOG' Subject: Re: D/DoS mitigation hardware/software needed. Stefan Fouant wrote on 11/01/10 14:45: If anyone is interested, I did pretty exhaustive research into the Service Provider marketplace last summer (before Verisign came out with their VIDN). I've got some slides which outline the costs, mitigation capacity, etc. of many different providers. The provider option isn't always the cheapest when compared to DIY factored in over a 3-5 year lifespan. If you can share, I think everyone would be interested. I am :)
Re: D/DoS mitigation hardware/software needed.
Firewalls do a good job of protecting servers, when properly configured, because they are designed exclusively for the task. Their CAM tables, realtime ASICs and low latencies are very much unlike the CPU-driven, interrupt-bound hardware and kernel-locking, multi-tasking software on a typical web server. IME it is a rare firewall that doesn't fail long, long after (that's after, not before) the hosts behind them would have otherwise gone belly-up. Then you need to get rid of that '90's antique web server and get something modern. When you say interrupt-bound hardware, all you are doing is showing that you're not familiar with modern servers and quality operating systems that are designed to mitigate things like DDoS attacks. Stateful filtering is to firewalls what interrupt-based packet processing is to web servers. Both are recipes for disaster. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: D/DoS mitigation hardware/software needed.
Then you need to get rid of that '90's antique web server and get something modern. When you say interrupt-bound hardware, all you are doing is showing that you're not familiar with modern servers and quality operating systems that are designed to mitigate things like DDoS attacks. Modern servers? IP is processed in the kernel on web servers, regardless of OS. Have you configured a kernel lately? Noticed there are ~3,000 lines in the Linux config file alone? _Lots_ of device drivers in there, which are interrupt driven and have to be timeshared. No servers I know do realtime processing (RT kernels don't) or process IP in ASICs. What configurations of Linux / BSD / Solaris / etc does web / email / ntp / sip / iptables / ipfw / ... and doesn't have issues with kernel locking? Test it on your own servers by mounting a damaged DVD on the root directory, and dd'ing it to /dev/null. Notice how the ATA/SATA/SCSI driver impacts the latency of everything on the system. How would you replicate that on a firmware and ASIC drive appliance? Roger Marquis
Re: D/DoS mitigation hardware/software needed.
Dobbins, Roland wrote: My employer's products don't compete with firewalls, they *protect* them; if anything, it's in my pecuniary interest to *encourage* firewall deployments, so said firewalls will fall down and need protection, heh. Nobody's disputing that Roland, or the fact that different specialized appliances will protect against different perimeter attacks. The only thing you've said that is being disputed is the the claim that a firewall under a DDoS type of attack will fail before a server under the same type of attack. I question this claim for several reasons. * because it doesn't correlate with my 22 years of experience in systems administration and 14 years in netops (including Yahoo netsecops where I did use IXIAs to compile stats on FreeBSD and Linux packet filtering), * it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment, * it doesn't correlate with server and firewall hardware and software designs, and last but not least, * because you have shown no objective evidence to support the claim. I did this kind of testing when I worked for the largest manufacturer of firewalls in the world Where then, can we find the results of your testing? Here's the thing; you're simply mistaken, and you hurl insults instead of listening to the multiple people on this thread who have vastly more large-scale Internet experience than you do and who concur with these prescriptions. Nobody has hurled insults in this thread other than yourself Roland. Shame on you for such disreputable tactics. To make the case you need more than repeated dismissal of requests for evidence and repeated unsupported claims of vast experience with failing servers and firewalls. We just need some actual statistics. Roger Marquis
Re: D/DoS mitigation hardware/software needed.
Then you need to get rid of that '90's antique web server and get something modern. When you say interrupt-bound hardware, all you are doing is showing that you're not familiar with modern servers and quality operating systems that are designed to mitigate things like DDoS attacks. Modern servers? IP is processed in the kernel on web servers, regardless of OS. Have you configured a kernel lately? Yes, pretty much every time I install a server. Noticed there are ~3,000 lines in the Linux config file alone? Well, that explains a lot. % wc -l /sys/i386/conf/WEBX4 324 /sys/i386/conf/WEBX4 I probably haven't noticed that there are ~3,000 lines in the Linux config file alone because I use a different OS; ~3,000 lines of config would just be another example of why I generally consider Linux to be a little broken. I can see why admins would be hesitant to challenge such a thing. _Lots_ of device drivers in there, which are interrupt driven and have to be timeshared. No servers I know do realtime processing (RT kernels don't) or process IP in ASICs. Roger, meet FreeBSD. FreeBSD, meet Roger. FreeBSD, would you please show Roger how IP is handled without excessive interrupts? % systat -vm (snipped from larger display) Interrupts 2208 total stray irq7 mux irq9 em5 irq5 85 ata0 irq14 mux irq11 fdc0 irq6 atkbd0 irq sio0 irq4 1995 clk irq0 128 rtc irq8 % netstat 1 input(Total) output packets errs bytespackets errs bytes colls 58991 0 54547321 58975 0 54523849 0 59492 0 58297208 59475 0 58388027 0 65828 0 62105928 65856 0 62081922 0 60257 0 56781863 60219 0 56809674 0 62547 0 61254034 62583 0 61231514 0 58188 9 55536734 58103 0 55560822 0 73870 0 70245952 73959 0 70223249 0 61436 0 58766122 61429 0 58786292 0 61390 0 59050710 61336 0 59029298 0 61447 0 58701312 61502 0 58725356 0 63934 0 60801413 63932 0 60777621 0 60187 0 56724030 60189 0 56751946 0 60247 0 55544082 60036 0 55522162 0 66472 0 63061572 66635 0 63033232 0 66415 0 62876955 66438 0 62854488 0 66612 0 63270235 66355 0 63335538 0 66020 0 60478426 66293 0 60454874 0 67696 0 63512069 67692 0 63534500 0 66342 0 60462142 66353 0 60439239 0 That's 60Kpps being handled with 2K interrupts per second. It'll be 2K interrupts per second at 0pps or 200Kpps or whatever. % ipfw l | wc -l 620 It's doing nontrivial amounts of firewalling while doing this. % top last pid: 83148; load averages: 0.31, 0.28, 0.23 up 459+08:00:24 12:00:33 51 processes: 3 running, 42 sleeping, 6 stopped CPU states: 14.8% user, 0.0% nice, 19.1% system, 13.3% interrupt, 52.7% idle % cat /var/run/dmesg.boot [...] CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2994.90-MHz 686-class CPU) Origin = GenuineIntel Id = 0xf41 Stepping = 1 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE [...] Ewww, but it *is* a 2004-vintage Pentium Prescott CPU on a legacy PCI mobo, so it is actually a little disadvantaged compared to modern hardware. What configurations of Linux / BSD / Solaris / etc does web / email / ntp / sip / iptables / ipfw / ... and doesn't have issues with kernel locking? That's like saying what cars cannot be crashed into a wall. A much better question is what combination of driver and vehicle can I get that significantly reduces the chances of my being involved in a crash. Driver is important because even the best vehicle can be driven into a wall; vehicle is important because even the best driver is severely limited by a decrepit old car. It's when you get a great driver in a great vehicle that you get the good results. Test it on your own servers by mounting a damaged DVD on the root directory, and dd'ing it to /dev/null. Notice how the ATA/SATA/SCSI driver impacts the latency of everything on the system. As soon as a remote attacker is able to insert a damaged DVD into one of my servers (maybe via specially crafted IP options in a TCP packet?), you will witness my posterior emit a large number of blocks of ceramic material (used in masonry construction). Until then, I am unfazed by this because it isn't particularly relevant to the discussion. I can cause excessive latency simply by switching off gear too. I *strongly* suggest you go and look over http://info.iet.unipi.it/~luigi/polling/ /and note its date/ before you compose any reply; device
Re: D/DoS mitigation hardware/software needed.
From someone who mostly lerks but has been in network engineering operations biz for 17 years, the only OS that seems to always keel over under a ddos and need a firewall is windows. Linux in its current incarnation can handle a substantially larger attack before needing mitigation by firewall type device. So in the end I believe its the environment dictates the use of products unless you have aformentioned windows os which for me has always necessitated a firewall. Manolo Sent from my BlackBerry -Original Message- From: Roger Marquis marq...@roble.com Date: Sun, 10 Jan 2010 08:55:13 To: nanog@nanog.org Subject: Re: D/DoS mitigation hardware/software needed. Dobbins, Roland wrote: My employer's products don't compete with firewalls, they *protect* them; if anything, it's in my pecuniary interest to *encourage* firewall deployments, so said firewalls will fall down and need protection, heh. Nobody's disputing that Roland, or the fact that different specialized appliances will protect against different perimeter attacks. The only thing you've said that is being disputed is the the claim that a firewall under a DDoS type of attack will fail before a server under the same type of attack. I question this claim for several reasons. * because it doesn't correlate with my 22 years of experience in systems administration and 14 years in netops (including Yahoo netsecops where I did use IXIAs to compile stats on FreeBSD and Linux packet filtering), * it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment, * it doesn't correlate with server and firewall hardware and software designs, and last but not least, * because you have shown no objective evidence to support the claim. I did this kind of testing when I worked for the largest manufacturer of firewalls in the world Where then, can we find the results of your testing? Here's the thing; you're simply mistaken, and you hurl insults instead of listening to the multiple people on this thread who have vastly more large-scale Internet experience than you do and who concur with these prescriptions. Nobody has hurled insults in this thread other than yourself Roland. Shame on you for such disreputable tactics. To make the case you need more than repeated dismissal of requests for evidence and repeated unsupported claims of vast experience with failing servers and firewalls. We just need some actual statistics. Roger Marquis
Re: D/DoS mitigation hardware/software needed.
On Sun, 10 Jan 2010 08:19:27 PST, Roger Marquis said: Then you need to get rid of that '90's antique web server and get something modern. When you say interrupt-bound hardware, all you are doing is showing that you're not familiar with modern servers and quality operating systems that are designed to mitigate things like DDoS attacks. Modern servers? IP is processed in the kernel on web servers, regardless of OS. Have you configured a kernel lately? Noticed there are ~3,000 lines in the Linux config file alone? _Lots_ of device drivers in there, which are interrupt driven and have to be timeshared. Yes, but all the fast network adapters are able to do a lot of stuff like interrupt coalescing so you don't need to take an interrupt on every packet. And have you configured a kernel lately is another red herring - yes, there are indeed be 4,533 lines in the current Fedora .config. But that's because that config turns on everything under the sun. I just checked, and my current kernel config has only 960 '=y' lines, and another 220 '=m' lines - and a large portion of those could easily be turned off. I have a minimal config file that comes in under 730 non-comment lines. No servers I know do realtime processing (RT kernels don't) or process IP in ASICs. That's because in general, processing the IP in an ASIC simply Does Not Work as well as you might hope. Alan Cox did a nice discussion of some of the issues here: http://lkml.indiana.edu/hypermail/linux/kernel/0307.1/2116.html One should read his last paragraph carefully, and note that what he wrote back in 2003 is still true today: http://www.internet2.edu/lsr/history.html What configurations of Linux / BSD / Solaris / etc does web / email / ntp / sip / iptables / ipfw / ... and doesn't have issues with kernel locking? So let me get this straight - you perceive a problem with locking inside the kernel, where if you're lucky the lock is in an already-hot cache line and your biggest worry is cache line ping-ponging, and if you're unlucky you actually have to go out the southbridge and hit main memory, at main memory access speeds. And to fix this, you're going to move one of the things contending for the lock off the CPU, so now every time the lock is contended, it has to go out through the PCI bridge to an external card? How the heck is that supposed to help? You're suggesting the same go talk to another card solution that the router vendors learned is the *last* thing you want to do - calling out to the supervisor card rather than handling it onboard the line card is guaranteed performance death. Test it on your own servers by mounting a damaged DVD on the root directory, and dd'ing it to /dev/null. Notice how the ATA/SATA/SCSI driver impacts the latency of everything on the system. How would you replicate that on a firmware and ASIC drive appliance? There's two little things you went astray on here: 1) I've in fact had to do this while doing data recovery. It doesn't do squat to the latency of anything that doesn't have to go through the same controller as the DVD. Everything else works just fine. Heck, it isn't even enough to cause audio playback skips (and those are noticeable even at the millisecond level). 2) Your latency hit is because the controller is *busy* while trying to re-read and error-correct a bad block. So yeah - trying to do I/O through a controller that's taking a several-second time-out dealing with bad media will cause a latency hit *for that I/O*. What's your point? pgpgtWs2YXuXm.pgp Description: PGP signature
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 11:55 PM, Roger Marquis wrote: The only thing you've said that is being disputed is the the claim that a firewall under a DDoS type of attack will fail before a server under the same type of attack. It's so obvious that well-crafted programmatically-generated attack traffic, if nothing else, will crowd out the good traffic that I'm just dumbfounded anyone thinks 'proof' of this is needed. Same thing for the fact that horizontally-scaled Web farm (with or without reverse caching proxies) will of necessity handle a great deal more TCP state than the biggest, firewall made to date. * because it doesn't correlate with my 22 years of experience in systems administration and 14 years in netops (including Yahoo netsecops where I did use IXIAs to compile stats on FreeBSD and Linux packet filtering), It doesn't correlate with my 25 years in the industry, a good portion of the last 15 years spent handling DDoS after DDoS after DDoS, during which the biggest, baddest firewalls choked and died over and over again, through multiple generations of said firewalls. Again, I was able to take down a hardware-based (for whatever value of 'hardware-based' is possible) firewall rated at 2gb/sec with 80kpps of traffic. * it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment, It correlates with my experience in large networks with geographically-dispersed IDCs with heterogeneous gear. * it doesn't correlate with server and firewall hardware and software designs, and last but not least, Which is a non-sequitur. * because you have shown no objective evidence to support the claim. I've my own broad subjective experience, and that of several other people who've commented on this thread have similar experiences. Since you haven't yet acquired this subjective experience, you can cause it to happen in a controlled test environment, should you so choose. Where then, can we find the results of your testing? The testing I did when I worked for the vendor in question is proprietary, as you can well surmise. You're free to do your own testing and confirm these assertions for yourself. Nobody has hurled insults in this thread other than yourself Roland. You accused me of acting in my own pecuniary interest, of trying to 'sell' things, *for no reason at all*. We just need some actual statistics. If you actually care about the truth of the matter, you're free to generate your own. If you read the RoK/USA DDoS preso to which I linked, you see the attack throughput and bandwidth metrics/host, and you also see where I noted multiple 'Web Application Firewalls', load-balancers, and so-called 'IPS' falling over as a result of those attacks. That gives you a range right there, along with some attack traffic characteristics, including average packet size. It makes no sense to put a stateful inspection device in front of servers, where *every single packet* is unsolicited, and therefore no state tracking is even possible in the first place. Stateless filters in hardware capable of mpps do a much better job, without the risk of falling over due to state-table exhaustion. Folks who've been unlucky enough to be subjected to significant DDoS attacks have run into this issue again and again and again. Perhaps you've simply been lucky; but one can't count on one's luck holding forever. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
From: Dobbins, Roland rdobb...@arbor.net Date: Sun, 10 Jan 2010 21:56:38 + On Jan 10, 2010, at 11:55 PM, Roger Marquis wrote: The only thing you've said that is being disputed is the the claim that a firewall under a DDoS type of attack will fail before a server under the same type of attack. It's so obvious that well-crafted programmatically-generated attack traffic, if nothing else, will crowd out the good traffic that I'm just dumbfounded anyone thinks 'proof' of this is needed. Same thing for the fact that horizontally-scaled Web farm (with or without reverse caching proxies) will of necessity handle a great deal more TCP state than the biggest, firewall made to date. * because it doesn't correlate with my 22 years of experience in systems administration and 14 years in netops (including Yahoo netsecops where I did use IXIAs to compile stats on FreeBSD and Linux packet filtering), It doesn't correlate with my 25 years in the industry, a good portion of the last 15 years spent handling DDoS after DDoS after DDoS, during which the biggest, baddest firewalls choked and died over and over again, through multiple generations of said firewalls. Again, I was able to take down a hardware-based (for whatever value of 'hardware-based' is possible) firewall rated at 2gb/sec with 80kpps of traffic. * it doesn't correlate with experience in large networks with multiple geographically disperse data centers where we did use Arbor, Cisco and Juniper equipment, It correlates with my experience in large networks with geographically-dispersed IDCs with heterogeneous gear. * it doesn't correlate with server and firewall hardware and software designs, and last but not least, Which is a non-sequitur. * because you have shown no objective evidence to support the claim. I've my own broad subjective experience, and that of several other people who've commented on this thread have similar experiences. Since you haven't yet acquired this subjective experience, you can cause it to happen in a controlled test environment, should you so choose. Where then, can we find the results of your testing? The testing I did when I worked for the vendor in question is proprietary, as you can well surmise. You're free to do your own testing and confirm these assertions for yourself. Nobody has hurled insults in this thread other than yourself Roland. You accused me of acting in my own pecuniary interest, of trying to 'sell' things, *for no reason at all*. We just need some actual statistics. If you actually care about the truth of the matter, you're free to generate your own. If you read the RoK/USA DDoS preso to which I linked, you see the attack throughput and bandwidth metrics/host, and you also see where I noted multiple 'Web Application Firewalls', load-balancers, and so-called 'IPS' falling over as a result of those attacks. That gives you a range right there, along with some attack traffic characteristics, including average packet size. It makes no sense to put a stateful inspection device in front of servers, where *every single packet* is unsolicited, and therefore no state tracking is even possible in the first place. Stateless filters in hardware capable of mpps do a much better job, without the risk of falling over due to state-table exhaustion. Folks who've been unlucky enough to be subjected to significant DDoS attacks have run into this issue again and again and again. Perhaps you've simply been lucky; but one can't count on one's luck holding forever. There is a culture that has developed a dogma that firewalls are THE solution. Be it DDOS or most any other security threat. Like many dogmas, it is ingrained into so many people that denial is essentially heresy. People simply know that a firewall is essential, so any contrary argument is obviously bogus or confused and must be denied. I used to work at the place that probably invented the stateful firewall and the folks who invented it became the priests of the firewall dogma and went forth and preached its value. Note that this predates DDOS by many years and that they did have some valid arguments. But the result was an army of security experts who scowled and marked the audit as FAILED if you did not front EVERYTHING with a firewall. I know of one case where an organization bought a firewall and programmed it to pass everything, just to fix an automatic failure of a security audit. Oddly, the auditor did not even look at who the firewall was configured. Simple presence of the box made him happy. I'm afraid that you are fighting a dogma that will only slowly be beaten into recognizing reality, but I appreciate your fighting the good fight. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: ober...@es.net Phone: +1 510 486-8634 Key
Re: D/DoS mitigation hardware/software needed.
Martin Hannigan wrote on 05/01/10 16:50: I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device - Outsource to service provider I want to add some stuff on this as I didn't see them with a quick check on the thread. Local solution always have a limit as bandwith will be exhausted before goin into your solution/network. Outsourced services have higher cost than Arbor but can handled more. Known leader of the clean-pipe solution is Prolexic http://www.prolexic.com/ Akamai and Verisign also tries to go on this market http://www.akamai.com/security (through CDN) http://www.verisign.com/internet-defense-network/index.html Best regards, Julien
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 11, 2010 at 12:26 AM, jul jul_...@yahoo.fr wrote: Martin Hannigan wrote on 05/01/10 16:50: I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device - Outsource to service provider I want to add some stuff on this as I didn't see them with a quick check on the thread. Local solution always have a limit as bandwith will be exhausted before goin into your solution/network. Outsourced services have higher cost than Arbor but can handled more. Do they? VerizonBusiness's solution was $3250US/month so ~$90USk over 2yrs. Arbor, I think, for a TMS + collectors was +100k. There are decent outsourced solutions, that move the problem out of your network, scrub traffic as requested, give you the ability to send traffic there on-demand (without calling the provider) and actually do work. All at a cost that's more than reasonable if your business depends upon the Internets. -chris
Re: D/DoS mitigation hardware/software needed.
On 2010-01-05 03:17, Tim Eberhard wrote: Kinda funny you state that Roland. I know of at least two very large carriers that uses Netscreens (and soon SRX's) for their DoS/DDoS mitigation. You mean Juniper SRX? The biggest box is a 5800, and it can handle up to 350k new sessions each second, up to maximum of 10 million (let's skip the fact that it's not that simple as it would look from the data sheet and there are major obstacles from reaching the numbers). 350kpps of TCP SYNs or whatever more wiser your botnet controller will generate, coming to your Internet pipe is really a small, very small DDoS. In terms of short packets like TCP SYN it's only around 179Mbit/s worth of bandwidth. Roland is right. Given finite resources to hold and process stateful information, the stateful device on a packet way to protected device is always vulnerable itself to become DDoSed. You can't discuss the logic of that, you can only throw more capable boxes and of course fail at some point. -- Everything will be okay in the end. | Łukasz Bromirski If it's not okay, it's not the end. | http://lukasz.bromirski.net
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Łukasz Bromirski [mailto:luk...@bromirski.net] Sent: Saturday, January 09, 2010 6:11 AM You mean Juniper SRX? The biggest box is a 5800, and it can handle up to 350k new sessions each second, up to maximum of 10 million (let's skip the fact that it's not that simple as it would look from the data sheet and there are major obstacles from reaching the numbers). With all due respect, I've been playing with the high end SRXs lately and I have to say I've been incredibly impressed with the performance... I recently did some performance testing on the SRX 5600s and I was able to consistently observe it instantiating upwards of 150k new TCP sessions per second. Does the SRX have some bugs... sure... that is to be expected with a box which by all means is still relatively bleeding edge. I'm fairly confident given a little time to stabilize the code, they will be able to fix some of the obstacles you are describing above... Having said that, I always laugh when I'm working with customers who have been DoSed and their response is Well, our firewall/load balancer has DDoS mitigation capabilities Almost every firewall or load balancer device I've worked with (Netscreen, SRX, Brocade, Fortinet) that had any sort of DoS mitigation features was extremely limited in its capability. Most only do session-based limiting towards a given destination IP, with the ultimate result being that they simply rate-limit the traffic towards that destination. This in itself ends up completing the attackers goal of denying service (even if just a subset) towards a given IP. And these types of features do nothing to assist with low-level attack traffic which require surgical mitigation, not to mention a host of other attack vectors. Firewalls do have their place in DDoS mitigation scenarios, but if used as the ultimate solution you're asking for trouble. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Jan 9, 2010, at 9:57 PM, Stefan Fouant wrote: Firewalls do have their place in DDoS mitigation scenarios, but if used as the ultimate solution you're asking for trouble. In my experience, their role is to fall over and die, without exception. I can't imagine what possible use a stateful firewall has being placed in front of servers under normal conditions, much less during a DDoS attack; it just doesn't make sense. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Dobbins, Roland [mailto:rdobb...@arbor.net] Sent: Saturday, January 09, 2010 10:03 AM On Jan 9, 2010, at 9:57 PM, Stefan Fouant wrote: Firewalls do have their place in DDoS mitigation scenarios, but if used as the ultimate solution you're asking for trouble. In my experience, their role is to fall over and die, without exception. I can't imagine what possible use a stateful firewall has being placed in front of servers under normal conditions, much less during a DDoS attack; it just doesn't make sense. See the earlier post - what I'm referring to here is more along the lines of stateless packet filters on upstream routers which can be triggered via Flowspec or similar mechanisms... I'm not disagreeing with you here on the other points and largely concur. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
We should circle up one day, I would love to provide you with some new experiences. There is no sense in chalk talking it, too often I also disagree with new ideas until I see them in action. Best regards, Jeff On Sat, Jan 9, 2010 at 10:03 AM, Dobbins, Roland rdobb...@arbor.net wrote: In my experience, their role is to fall over and die, without exception. I can't imagine what possible use a stateful firewall has being placed in front of servers under normal conditions, much less during a DDoS attack; it just doesn't make sense. -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Follow us on Twitter at http://twitter.com/ddosprotection to find out about news, promotions, and (gasp!) system outages which are updated in real time. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to protect your booty.
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 12:57 AM, Jeffrey Lyon wrote: I would love to provide you with some new experiences. I get new experiences of this type and plenty of new ideas every day, thanks. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Dobbins, Roland wrote: Firewalls do have their place in DDoS mitigation scenarios, but if used as the ultimate solution you're asking for trouble. In my experience, their role is to fall over and die, without exception. That hasn't been my experience but then I'm not selling anything that might have a lower ROI than firewalls, in small to mid-sized installations. I can't imagine what possible use a stateful firewall has being placed in front of servers under normal conditions, much less during a DDoS attack; it just doesn't make sense. Firewalls are not designed to mitigate large scale DDoS, unlike Arbors, but they do a damn good job of mitigating small scale attacks of all kinds including DDoS. Firewalls actually do a better job for small to medium sites whereas you need an Arbor-like solution for large scale server farms. Firewalls do a good job of protecting servers, when properly configured, because they are designed exclusively for the task. Their CAM tables, realtime ASICs and low latencies are very much unlike the CPU-driven, interrupt-bound hardware and kernel-locking, multi-tasking software on a typical web server. IME it is a rare firewall that doesn't fail long, long after (that's after, not before) the hosts behind them would have otherwise gone belly-up. Rebooting a hosed firewall is also considerably easier than repairing corrupt database tables, cleaning full log partitions, identifying zombie processes, and closing their open file handles. Perhaps a rhetorical question but, does systems administration or operations staff agree with netop's assertion they 'don't need no stinking firewall'? Roger Marquis
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 9:03 AM, Roger Marquis wrote: That hasn't been my experience but then I'm not selling anything that might have a lower ROI than firewalls, in small to mid-sized installations. I loudly evinced this position when I worked for the world's largest firewall vendor, so that dog won't hunt, sorry. Think about it; firewalls go down under DDoS *much more quickly than the hosts themselves*; Arbor and other vendor's IDMSes protect many, many firewalls unwisely deployed in front of servers, worldwide. Were I that sort of person (and I'm not, ask anyone who knows me), it's in my naked commercial interest to *promote* firewall deployments, so that *more* sites will go down more easily and require IDMSes, heh. Firewalls are not designed to mitigate large scale DDoS, unlike Arbors, but they do a damn good job of mitigating small scale attacks of all kinds including DDoS. Not been my experience at all - quite the opposite. Firewalls actually do a better job for small to medium sites whereas you need an Arbor-like solution for large scale server farms. No, S/RTBH and/or flow-spec are a much better answer for sites which don't need IDMS, read the thread. And they essentially cost nothing from a CAPEX perspective, and little from an OPEX perspective, as they leverage the existing network infrastructure. Firewalls do a good job of protecting servers, when properly configured, because they are designed exclusively for the task. No, they don't, and no, they aren't. Their CAM tables, realtime ASICs and low latencies are very much unlike the CPU-driven, interrupt-bound hardware and kernel-locking, multi-tasking software on a typical web server. IME it is a rare firewall that doesn't fail long, long after (that's after, not before) the hosts behind them would have otherwise gone belly-up. Completely incorrect on all counts. Sales propaganda regurgitated as gospel. Rebooting a hosed firewall is also considerably easier than repairing corrupt database tables, cleaning full log partitions, identifying zombie processes, and closing their open file handles. Properly-designed server installations don't have these problems. Firewalls don't help, either - they just go down. Perhaps a rhetorical question but, does systems administration or operations staff agree with netop's assertion they 'don't need no stinking firewall'? I've been a sysadmin, thanks. How about you? You can assert that the sun rises in the West all you like, but that doesn't make it true. All the assertions you've made above are 100% incorrect, as borne out by the real-world operational experiences of multiple people who've commented on this thread, not just me. I've worked inside the sausage factory, FYI, and am quite familiar with how modern firewalls function, what they can do, and their limitations. And they've no place in front of servers, period. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Dobbins, Roland wrote: Firewalls are not designed to mitigate large scale DDoS, unlike Arbors, but they do a damn good job of mitigating small scale attacks of all kinds including DDoS. Not been my experience at all - quite the opposite. Ok, I'll bite. What firewalls are you referring to? Their CAM tables, realtime ASICs and low latencies are very much unlike the CPU-driven, interrupt-bound hardware and kernel-locking, multi-tasking software on a typical web server. IME it is a rare firewall that doesn't fail long, long after (that's after, not before) the hosts behind them would have otherwise gone belly-up. Completely incorrect on all counts. So then you're talking about CPU-driven firewalls, without ASICs e.g., consumer-level gear? Well, that would explain why you think they fail before the servers behind them. I've been a sysadmin Have you noticed how easily Drupal servers go down with corrupt MyISAM tables? How would S/RTBH and/or flow-spec protect against that? Roger Marquis
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 10:05 AM, Roger Marquis wrote: Ok, I'll bite. What firewalls are you referring to? Hardware-based commercial firewalls from the major vendors, open-source/DIY, and anything in between. All stateful firewalls ever made, period (as discussed previously in the thread). So then you're talking about CPU-driven firewalls, without ASICs e.g., consumer-level gear? Well, that would explain why you think they fail before the servers behind them. You obviously haven't read the thread. No, I'm not talking about little firewalls, and no, I don't 'think' anything - I *know* it, because I've seen it over and over again, including during my tenure at the largest commercial firewall vendor in the world. See here for a high-profile example: http://files.me.com/roland.dobbins/k54qkv I've personally choked a hardware-based firewall rated at 2gb/sec with only 80kpps of traffic from an old, PowerPC-based PowerBook, for example. And again, as noted repeatedly in the thread, all that's required to effectively DDoS servers behind firewalls is to programmatically generate well-formed, completely valid traffic which passes all the firewall rules/inspectors/what-have-you - enough to 'crowd out' legit traffic from legit users. I strongly suggest reading the thread before commenting. Have you noticed how easily Drupal servers go down with corrupt MyISAM tables? How would S/RTBH and/or flow-spec protect against that? We're talking about DDoS mitigation in order to keep the servers up and running, so that they don't go down ungracefully and corrupt anything in the first place. Placing a stateful inspection device in a topological position where no stateful inspection is possible due to every incoming packet being unsolicited makes zero sense whatsoever from an architectural standpoint, even without going into implementation-specific details. Once again - read the thread. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Sat, Jan 9, 2010 at 10:21 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 10, 2010, at 10:05 AM, Roger Marquis wrote: Have you noticed how easily Drupal servers go down with corrupt MyISAM tables? How would S/RTBH and/or flow-spec protect against that? We're talking about DDoS mitigation in order to keep the servers up and running, so that they don't go down ungracefully and corrupt anything in the first place. have you noticed how putting your DB and WEB server on the same hardware is a bad plan? separate the portions of the pie... only let the attack break the minimal portion of your deployment. Use the right tool in the right place. -chris
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 10:33 AM, Christopher Morrow wrote: separate the portions of the pie... only let the attack break the minimal portion of your deployment. Use the right tool in the right place. An excellent point. A Web front-end server should be that - merely the front-end. Situationally-appropriate functional separation is a key architectural principle. Combine that with the measures outlined in http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html, coupled with a functionally-separated, bulkheaded DNS infrastructure pictured in http://files.me.com/roland.dobbins/m4g34u, and one will end up with a much more robust, scalable, and defensible system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Dobbins, Roland wrote: See here for a high-profile example: http://files.me.com/roland.dobbins/k54qkv Reads like a sales pitch to me. No apples to apples comparisons, nothing like an ANOVA of PPS, payload sizes, and other vectors across different types of border defenses. Your presentation makes a good case for Arbor-type defenses, against a certain type of attack, but it doesn't make the case you're referring to. What would convince me is an IXIA on a subnet with ten hosts running a db-bound LAMP stack. Plot the failure points under different loads. Then add an ASA or Netscreen and see what fails under the same loads. That would be an objective measure, unlike what has been offered as evidence in this thread so far. Placing a stateful inspection device in a topological position where no stateful inspection is possible due to every incoming packet being unsolicited makes zero sense whatsoever from an architectural standpoint, even without going into implementation-specific details. Which is basically claiming that the general purpose web server, running multiple applications, is more capable of inspecting every incoming packet than hardware specifically designed for the task and doing only the task it was designed for. Christopher Morrow wrote: have you noticed how putting your DB and WEB server on the same hardware is a bad plan? While often true this is entirely tangental to the thread. Roger Marquis
RE: D/DoS mitigation hardware/software needed.
Firewalls are not designed to mitigate large scale DDoS, Generally speaking, if it didn't being the firewall to its knees, it wasn't a DoS. It was just sort of an annoying attempt at a DoS. I think that more or less the definition of a DoS is one that exploits the resource limitations of the firewall to deny service to everything behind it. The ultimate DoS, though, is simply filling the pipe with traffic from legitimate data transfer requests. Nothing you are going to do is going to mitigate that because to stop it you have to DoS yourself. Imagine thousands of requests per second from all around the internet for a legitimate URL. How do you use a firewall to separate the wheat from the chaff? So let's say you have some client software that you want people to download. Suddenly you are getting more download requests than you can handle. Nobody is flooding you with syn or icmp packets. They are sending a single packet (a legitimate URL) that results in you sending thousands of packets to real IP addresses that are simply copying the traffic to what amounts to /dev/null. Now when your download server gets slow, things get worse because connections begin to take longer to clear. The kernel on the web server is able to handle the tcp/ip setup fairly quickly but getting the file actually shipped out takes time. As connections build up on the firewall, it finally reaches a point where it is out of RAM in storing all those NAT translations and connection state. Now you start noticing that services not under attack are starting to slow down because the firewall has to sort through an increasingly large connection table when doing stateful inspection of traffic going to other services. All the while, there really isn't anything the firewall can do to mitigate the traffic because it is all correct and legitimate. Basically you are being Slashdotted or experiencing the Drudge Effect but in this case you are being botnetted. If you have the server capacity to keep up, now your outbound pipe to the Internet is filling up, you are dropping packets, TCP/IP connections begin to back off, connections back up even more and at some point the firewall just gives up by failing over to the secondary, which then promptly fails back to the primary and you bounce back and forth in that state for a while and then finally it just gets hung someplace and the whole thing is stuck. And during the entire incident there was no illegal traffic that your firewall could have done a thing to block. Oh, and rate limiting connections isn't going to fix things either unless you can do it on a per URL basis. Maybe the rate of requests for /really-big-file.tgz that clogs your system is way different than the rate of requests for /somewhat-smaller-file.tgz or /index.html
Re: D/DoS mitigation hardware/software needed.
On Jan 10, 2010, at 1:27 PM, Roger Marquis wrote: Reads like a sales pitch to me. My employer's products don't compete with firewalls, they *protect* them; if anything, it's in my pecuniary interest to *encourage* firewall deployments, so said firewalls will fall down and need protection, heh. Teaching people how to design their server farms, harden their network infrastructure, and deploy S/RTBH and flow-spec isn't selling anything. Only someone with ulterior motives would claim otherwise. This isn't 'selling' anything, either: http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html So, this line of attack falls flat, and merely comes across as unjustified, uninformed, foolish and petty. Your presentation makes a good case for Arbor-type defenses, against a certain type of attack, but it doesn't make the case you're referring to. S/RTBH and flow-spec aren't 'Arbor-type defenses', and I had a long track record of making the case for all of these things for many years before I ever worked for Arbor. What would convince me is an IXIA on a subnet with ten hosts running a db-bound LAMP stack. Plot the failure points under different loads. Then add an ASA or Netscreen and see what fails under the same loads. Then hop to it. I did this kind of testing when I worked for the largest manufacturer of firewalls in the world, so I've no need to repeat it. Which is basically claiming that the general purpose web server, running multiple applications, is more capable of inspecting every incoming packet than hardware specifically designed for the task and doing only the task it was designed for. Properly tuned, yes. Here's the thing; you're simply mistaken, and you hurl insults instead of listening to the multiple people on this thread who have vastly more large-scale Internet experience than you do and who concur with these prescriptions. That's your prerogative; and it's my prerogative to grow tired repeating the same points which have already been made earlier in this and other threads, when they fall on biased, deaf ears. If you choose not to read and understand and learn from the broader experiences of others, that's up to you. I'm done. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
At 13:19 05/01/2010 +, Rob Shakir wrote: If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! In that case, how do you run your current service: http://www.vialtus.com/en/Solutions/Hosting-and-Datacentre-Services/Security-Solutions/Distributed-Denial-of-Service-Protection.aspx -Hank Kind regards, Rob -- Rob Shakir r...@eng.gxn.net Network Development EngineerGX Networks/Vialtus Solutions ddi: +44208 587 6077mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http://www.vialtus.com/disclaimer.html
Re: D/DoS mitigation hardware/software needed.
On Wed, 2010-01-06 at 17:00 +0200, Hank Nussbacher wrote: In that case, how do you run your current service: http://www.vialtus.com/en/Solutions/Hosting-and-Datacentre-Services/Security-Solutions/Distributed-Denial-of-Service-Protection.aspx It says how, right on that page. Not Arbor. Graeme (ex PIPEX employee who had first hand experience of just how good the aforementioned Cisco Guard kit was in production)
Re: D/DoS mitigation hardware/software needed.
* Adrian Chadd: Has anyone deployed a DDoS distributed enough to inject ETOOMANY routes into the hardware forwarding tables of routers? I think this is called injecting the BGP table into your IGP, and quite a few folks have done it. Nobody claimed that it was an act of sabotage, though. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
Re: D/DoS mitigation hardware/software needed.
I know of several companies, with large websites, that used code reviews as at least one way they met this DSS requirement. So, erm, empirically denied. The PCI DSS does not require code review of the software running in COTS equipment, nor of underlying OS's or applications. It requires a code review of the application code that is inside PCI scope. In general, this means the code you write to run your website is the maximum scope of this requirement. Plenty of companies allow code reviews for security and other purposes, and with good reason. There exist entire practices in IT security firms dedicated to performing code reviews, and they appear to be growing. Also, the PCI security council allows people to use automated code auditing tools (such as fortify), performing a manual application assessment- which plenty of firms will let you pay them to do, or even to use an automated web application security scanners. Several vendors of Vulnerability Assessment tools that meet this spec are available. I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%. To make the operational content clear- if someone tells you that you need to buy a Web Application Firewall to meet PCI requirements (process credit cards), be aware that is not the only option. I'd recommend you choose the option that is most likely to genuinely improve the security of your infrastructure and business, which may well be a WAF. --D On Mon, Jan 4, 2010 at 11:54 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: PCI DSS does not require a Web application firewall. http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1313797,00.html Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: D/DoS mitigation hardware/software needed.
Adrian Chadd wrote: On Tue, Jan 05, 2010, Dobbins, Roland wrote: None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions. But as you said, they're willing to sell them to you. Then claim that the traffic you're receiving is out of profile. :) (I'm not jaded about this, oh no..) ...so, out of curiosity, which one did you buy? ;) Steve
Re: D/DoS mitigation hardware/software needed.
Comments inline. To reiterate- my entire point is that stateful firewalls are at least sometimes useful in front of large websites. Something of a veer off of the original topic, but not an at all useless exercise IMHO. On Mon, Jan 4, 2010 at 11:47 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless. Stateful policies help protect against misbehaving software, operations error, malicious internal activity, external attackers and trojans/viruses/bots making outbound connections to arbitrary ports anywhere on the Internet. Seems to have a point to me. See my previous point about blocking outbound connections. If you allow internal hosts to make connections out to any port on any ip, then perhaps this point is invalid. But lots of people _don't_ allow outbound from any to any/any, and thats a good thing in my book. As for performance, you can get .9mpps stateful firewalls for reasonable prices, and gear exists that can scale up to 15mpps. * Location for IDS/IDP. Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh). The gear I'm talking about above supports IDP on the same platform. If I am going to deploy an IDP, why wouldn't I choose one that is flow based so it blocks attacks rather than send out panicked RST's in response to attacks, and which has a stateful firewall included (yes, any flow based IDP is essentially a stateful firewall, and I suspect that a well managed IDP with customized rules is similar in functionality to what WAF's do out of the box. In fact one vendors WAF product functionality pre-trained out of the box is snort rules) * Connection cleanup, re-assembling fragments, etc. Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers. Huh, I would have sworn I've seen fragment based attacks on certain linux kernels and Windows systems. I'm sure it won't ever happen again. Load-balancers may be a good place for this, but there are good reasons that you would prefer to have this taken care of as close to ingress as possible. For example, if you want to compare packet flows of a connection on either side of a load-balancer, it is easier to do that if the incoming connections have already been cleaned up. * SYN flood protection, etc. Firewalls simply don't handle this well, marketing claims aside. They crash and burn. I believe that to have been true in the not so recent past, I don't think it is true of more recent equipment. * Single choke point to block incoming traffic deemed undesirable. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. A firewall with easier to use CLI, web UI, management station and API is a much easier way to implement this and offers lots of control for time-of-day based ACL's, etc. etc. I like screening routers, and in a DOS situation I would recommend blocking as close to ingress as possible (preferably outside your own network!) but for typical blocking of nuisance connections, full featured firewalls that can be integrated with a configuration change tracking tool are a better way to go. And what about when the way of identifying undesirable traffic is something other than the source/dest ip/port such as a URI? (more of an argument for an IDP than a simply stateful firewall) * Single log point for inbound connections for analysis and auditing requirements. Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement. Really? How does netflow show me that a packet was dropped because of rule X? Or permitted because of rule Y? It's great at showing me data about the flow that was extant, but it doesn't tell me the context of the enforcement performed on it. I can assure you that there is a use in looking over denied packet logs. Firewall logs provide _more_ context than netflow. For DOS detection I get that netflow is probably the best tool to use- not arguing that- you made the statement that stateful firewalls are always useless in front of large websites/webservers. I am saying that they are usefull, at least some of the time, and that it is common for them to be deployed in that manner. * Allows outbound traffic enforcement. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. So you advocate allowing any packet with ACK set
Re: D/DoS mitigation hardware/software needed.
My basis for this is discussions with PCI assessors from multiple firms that perform large numbers of assessments per year. Next time I run into some, I'll ask to see if the usage has increased, its been a few months since I asked this of any of them. --D On Tue, Jan 5, 2010 at 1:02 AM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 3:58 PM, Darren Bolding wrote: I believe their is strong evidence that the use of web application firewalls to meet this DSS requirement is smaller than you might think. I would not be surprised if it was significantly less than 50%- perhaps 20%. This directly contradicts my experience working for vendor of such products, FWIW. But I hope this is indeed the case, as it will lead to higher availability for organizations which go this route! --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- -- Darren Bolding -- -- dar...@bolding.org --
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 5:04 PM, Darren Bolding wrote: To reiterate- my entire point is that stateful firewalls are at least sometimes useful in front of large websites. I understand completely; I simply disagree, stating my reasons for doing so in detail inline. It's my contention that under no circumstances are stateful firewalls *ever* useful in front of *any* Web server, at *any* time, in *any* deployment scenario, either public or private. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
(Resent, sorry for multiple copies -- I messed up from From: address) On 5 Jan 2010, at 06:16, Stefan Fouant wrote: That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? Well of course, moving to Arbor haha... eased in part by a little Cisco initiative called Clean Pipes 2.0 :) Is this really true? I've seen the white paper, I've been told that the this is the best way forward from the Guard, but I must say that I'm not yet totally convinced. The Guard product was something that can be separated from the Cisco Detection approach, i.e. one can activate the Guard via a means that did not necessarily involve the Detectors being the source of the activation, this doesn't seem to be true for the Arbor alternative (I believe that the TMS requires registering against the rest of the PeakFlow platform). The other thing that we noted relating to the platform is that there's nothing really new in the TMS (other than of course, much increased scrubbing rates!) compared to the Guard. There doesn't appear to be any direct comparison to the 'strong' scrubbing mode that the Cisco Guard implemented - whereby the device would proxy a bunch of traffic. If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! Kind regards, Rob -- Rob Shakir r...@eng.gxn.net Network Development EngineerGX Networks/Vialtus Solutions ddi: +44208 587 6077mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http://www.vialtus.com/disclaimer.html
Re: D/DoS mitigation hardware/software needed.
My somewhat educated opinion on the matter is that appliance developers want to sit on the edge and see all your traffic merely to protect their own interests and market share. NS-5000s have been good to us for bulk filtering and we rely on appliances for more intelligent inspection. Dollar for dollar NS is substantially more robust in my experience. Best regards, Jeff On Jan 5, 2010 9:46 AM, Rob Shakir r...@eng.gxn.net wrote: (Resent, sorry for multiple copies -- I messed up from From: address) On 5 Jan 2010, at 06:16, Stefan Fouant wrote: That said, what are all those ISPs doing now th... Is this really true? I've seen the white paper, I've been told that the this is the best way forward from the Guard, but I must say that I'm not yet totally convinced. The Guard product was something that can be separated from the Cisco Detection approach, i.e. one can activate the Guard via a means that did not necessarily involve the Detectors being the source of the activation, this doesn't seem to be true for the Arbor alternative (I believe that the TMS requires registering against the rest of the PeakFlow platform). The other thing that we noted relating to the platform is that there's nothing really new in the TMS (other than of course, much increased scrubbing rates!) compared to the Guard. There doesn't appear to be any direct comparison to the 'strong' scrubbing mode that the Cisco Guard implemented - whereby the device would proxy a bunch of traffic. If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! Kind regards, Rob -- Rob Shakir r...@eng.gxn.net Network Development EngineerGX Networks/Vialtus Solutions ddi: +44208 587 6077mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http://www.vialtus.com/disclaimer.html
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 9:59 PM, Jeffrey Lyon wrote: My somewhat educated opinion on the matter is that appliance developers want to sit on the edge and see all your traffic merely to protect their own interests and market share. This isn't generally a smart approach; the value of providing maximum topological coverage and the ability to oversubscribe the system by moving inwards a bit from the edge provides clear advantages, in most circumstances. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 4:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device - Outsource to service provider Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. How often are you getting DDoS'd? The financials of using a managed service provider vs. buy-all-your-own-grrovy-stuff can be fairly compelling especially if the amount of DDoS you experience is almost nil. Re: Arbor. I don't have any recent experience, but they've been around for a long time, have a very experienced team that understands ISP and enterprise and the product is mature. Hard to go wrong if you can justify the costs. YMMV. Best, -M -- Martin Hannigan mar...@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
Re: D/DoS mitigation hardware/software needed.
I looked at one of the suggested out-sourced providers. Based on a sample size of 1, the mitigating mechanisms are DNS redirection and BGP/tunneling. While both of these solutions may be useful for an end-user (even large ones), I don't see them fitting in an SP environment. If something goes wrong, I want my own, local, big-red button. Rick On Tue, Jan 5, 2010 at 7:50 AM, Martin Hannigan mar...@theicelandguy.comwrote: On Mon, Jan 4, 2010 at 4:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device - Outsource to service provider Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. How often are you getting DDoS'd? The financials of using a managed service provider vs. buy-all-your-own-grrovy-stuff can be fairly compelling especially if the amount of DDoS you experience is almost nil. Re: Arbor. I don't have any recent experience, but they've been around for a long time, have a very experienced team that understands ISP and enterprise and the product is mature. Hard to go wrong if you can justify the costs. YMMV. Best, -M -- Martin Hannigan mar...@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 9:44 PM, Rob Shakir wrote: If you're an SP who has some existing NetFlow solution, and don't really justify a spend for traffic intelligence within your network (or have something home-grown), is there an alternative scrubber that one might be able to use in a more standalone deployment that can approach the filtering levels of the Arbor kit? One thing folks can do is to implement S/RTBH and/or flow-spec at the edges, then take some Intel boxes, throw some high-performance network cards in them, along with Snort-inline, and put them in a scrubbing center, making use of BGP diversion/re-injection to get traffic into them on an as-needed basis. Don't make use of the useless bidirectional/stateful 'IPS' signatures, but do manual filtering on a case-by-case basis, unidirectionally. Below that, put a layer of WCCPv2-clustered Squid proxies to provide additional layer-7 filtering capabilities for Web-based traffic. Below that, re-inject the scrubbed traffic and send it on its way via one's redirection mechanism of choice to the destination servers. Does this 'poor man's IDMS' do everything and scale to the degree of the various commercial systems? No, of course not. But it's a way to get into layer-4-plus dynamic DDoS mitigation in a relatively economical way (at least from a capex perspective); for some folks, this type of solution may prove sufficient to needs, keeping in mind the limitations of this approach and the systems integration/support burden involved, of course. And even if/when it's clear more advanced capabilities are needed, starting out this way, even in a limited PoC, provides valuable operational experience with the diversion/re-injection model which will prove useful in evaluating more advanced commercial systems an an appropriate juncture. I should probably point out that we only really started our conversation with Arbor within the last month or so, so there are perhaps details relating to this that I've missed. I'd be happy to be corrected! There's actually quite a bit more, but short of answering specific questions in an operational context, this kind of vendor-specific discussion is probably well beyond what vendor employees like me (these strictures don't apply to anyone else, mind) can and should in all propriety participate in on nanog-l; please feel free to reach out 1:1 to the Arbor folks you've already been talking with to discuss further, and I'm happy to help out 1:1, as well. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, 5 Jan 2010 04:20:51 + Dobbins, Roland rdobb...@arbor.net wrote: S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed. I still wish we would have had something like Bellovin's Pushback implemented as a separate protocol rather than flow-spec over BGP, but having lost that battle we have been playing around with a (free) community, but vetted participant, flow-spec over BGP service if folks are interested in trying it out. Just shoot me note offline. You need an ASN, a flow-spec capable router and must be a verifiable admin/sec contact for said ASN (whatever that means :-). Basic idea is for folks who want to receive one or more sets of flow-spec feeds and/or inject things they want others to filter on (limited to your own routes) you can do so. No need for direct peering and like you say Roland, many networks already have all they need to start doing these sorts of things. Please note, we realize there are a variety of issues in implementing this sort of thing, but if we can find a way to make it trustworthy and workable, that is why we're here. Those not familiar with flow-spec can read up: http://tools.ietf.org/html/rfc5575 In a nutshell, distributed router filters via BGP. John
Re: D/DoS mitigation hardware/software needed.
We have substantial direct experience with both RioRey and IntruGuard. RR is more plug and play while IG is more robust but both are great. Use a robust firewall such as a Netscreen in front of your mitigation tool. Best regards, Jeff On Mon, Jan 4, 2010 at 4:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Follow us on Twitter at http://twitter.com/ddosprotection to find out about news, promotions, and (gasp!) system outages which are updated in real time. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to protect your booty.
RE: D/DoS mitigation hardware/software needed.
Rick, If you pass me your contact info I can forward it to our Arbor Sales guy who can get in touch with you. I been pretty impressed by Arbor so far. Thanks, Raj Singh -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Monday, January 04, 2010 1:20 PM To: NANOG Subject: D/DoS mitigation hardware/software needed. Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
Several responses already, and Arbor has poked their head up. I'm going to start there and keep the other suggestions at-hand. Thanks, On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
Ask them if they'd come down to $10 - 20k for a full featured model and they might make two sales, although I doubt it unfortunately. Best regards, Jeff On Mon, Jan 4, 2010 at 4:59 PM, Rick Ernst na...@shreddedmail.com wrote: Several responses already, and Arbor has poked their head up. I'm going to start there and keep the other suggestions at-hand. Thanks, On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick -- Jeffrey Lyon, Leadership Team jeffrey.l...@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Follow us on Twitter at http://twitter.com/ddosprotection to find out about news, promotions, and (gasp!) system outages which are updated in real time. Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 - 21 to find out how to protect your booty.
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Kinda funny you state that Roland. I know of at least two very large carriers that uses Netscreens (and soon SRX's) for their DoS/DDoS mitigation. State table exhaustion on the netscreen platform is something that is very easy to protect against with some protections and hasn't been a problem for years. If you can fill up a session table on a higher end SRX then I would be very very impressed. I would argue that firewalls place is in fact directly infront of servers and load balancers to protect them. On Mon, Jan 4, 2010 at 8:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
If you want to recreate D/DoS from captures (for testing purposes) you might want to check out: http://www.pcapr.net/dos This lets you validate how your mitigation solutions are holding up. K. On Mon, Jan 4, 2010 at 1:19 PM, Rick Ernst na...@shreddedmail.com wrote: Looking for D/DoS mitigation solutions. I've seen Arbor Networks mentioned several times but they haven't been responsive to literature requests (hint, if anybody from Arbor is looking...). Our current upstream is 3x GigE from 3 different providers, each landing on their own BGP endpoint feeding a route-reflector core. I see two possible solutions: - Netflow/sFlow/***Flow feeding a BGP RTBH - Inline device Netflow can lag a bit in detection. I'd be concerned that inline devices add an additional point of failure. I'm worried about both failing-open (e.g. network outage) and false-positives. My current system is a home-grown NetFlow parser that spits out syslog to our NOC to investigate potential attacks and manually enter them into our RTBH. Any suggestions other than Arbor? Any other mechanisms being used? My idea is to quash the immediate problem and work additional mitigation with upstreams if needed. I could probably add some automation to my NetFlow/RTBH setup, but I still need to worry about false-positives. I'd rather somebody else do the hard work of finding the various edge-cases. Thanks, Rick
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 9:17 AM, Tim Eberhard wrote: I would argue that firewalls place is in fact directly infront of servers and load balancers to protect them. The very idea of placing a stateful firewall in front of a Web/DNS/email/etc. server, in which *every single incoming packet is unsolicited, and therefore, leaves no state to be inspected in the first place*, is absurd. There is simply no valid argument for doing so. Hardening the OS/apps/services, combined with stateless ACLs in hardware which can handle mpps, is the way to enforce policy. In fact, the idea is such a poor one that one of the major firewall vendors came out with a 'stateful inspection bypass' feature - the idea being that, you buy their 10gb/sec, $100K-plus stateful firewall, stick it in front of servers, and then . . . disable the stateful inspection, heh. ; None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 05, 2010, Dobbins, Roland wrote: None of the large, well-known Web properties on the Internet today - at least, the ones which stay up and running, heh - have stateful firewalls in front of them. Including prominent vendors of said stateful firewall solutions. But as you said, they're willing to sell them to you. Then claim that the traffic you're receiving is out of profile. :) (I'm not jaded about this, oh no..) Adrian
Re: D/DoS mitigation hardware/software needed.
We have such a configuration in progress, it works great without any of the issues you're proposing. Jeff On Jan 4, 2010 9:09 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in fro... Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: We have such a configuration in progress, it works great without any of the issues you're proposing. So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:14 AM, Dobbins, Roland wrote: If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection. That should read 'state-table exhaustion', apologies for the typo. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:06 AM, Jeffrey Lyon wrote: We have such a configuration in progress, it works great without any of the issues you're proposing. Then you aren't testing it to destruction, heh. ; If it's a stateful firewall, and state-tracking is turned on, it's quite possible to craft sufficient pathological traffic which conforms to the firewall policies and yet which leads to state-table inspection. And the stateful firewall serves no purpose in front of servers, in which *every incoming packet* is unsolicited. Far more sensible to enforce policy in stateless ACLs in ASIC-based router/switch hardware. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
Two more options. And for Netflow device - read that to mean Arbor or its competitors. 5 Ditch the stateful firewall and exclusively use a netflow device 6. Outsource to a hosted DDoS mitigation service (Prolexic etc) On Tue, Jan 5, 2010 at 8:43 AM, Suresh Ramasubramanian ops.li...@gmail.com wrote: Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people]
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote: 5 Ditch the stateful firewall and exclusively use a netflow device NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play. PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers. It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 9:18 PM, jim deleskie deles...@gmail.com wrote: What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. not to pile on, but... +1 to roland here as well. I've seen more than enough folks put in a 'firewall' in front of their 'server' (say a mail server) and then watch that die long before the rest of the system did. Now, if you have equipment capable today of doing a few million session creates/second and you feel comfortable that you can keep track of how attacks grow vs your capacity stays the same and move ahead of the curve well enough, then... by all means do as you want :) There's a cost analysis which Roland sidestepped here as well, state-tracking at the rates required is expensive, as compared to relatively simple acls in hardware with no state on the upstream router. Spend where it matters, and make sure you understand where the failure points are that you place into your network. -chris -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic. 2. Indeed. 3. Sort of, such devices are downstream for extremely valid reasons I won't get into now. 4. Indeed, were equipped to handle substantially higher than 150kpps. I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though. Best regards, Jeff Best regards, Jeff On Jan 4, 2010 10:14 PM, Suresh Ramasubramanian ops.li...@gmail.com wrote: On Tue, Jan 5, 2010 at 8:36 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: We have such a c... So .. this is interesting. The firewall would have to frontend your mail / web / whatever application .. and if something goes beyond the firewall's rated capacity (100k ++ - maybe nearly 150..175k connections per second for a high end firewall), the firewall falls over. And even before that, there's the risk of whatever application you're protecting getting pounded flat if your firewall passes even a small percentage of this traffic. Do you - 1. Have (say) two firewalls in HA config? 2. Back your firewall with routing based measures, S/RTBH, blackhole communities your upstream offers, etc [the standard nspsec bootcamp stuff] 3. Simply back the firewall with a netflow based device? 4. Estimate that the risk of a DDoS that exceeds your firewall's rated capacity is extremely low? [and yes, 150k ++ connections per second ddos is going to be massive, and relatively rare for most people] --srs -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
A lot of this has to do with scaling the environment. I've had plenty of asa's and even netscreens fall over from state-table and session limitations. I've also seen a hosts fill up the connection table prior to a firewall being affected. I'm not familiar with the specs and anyone can chime in, but the newer variety of SRX's, I believe implement more in hardware as line-rate routers do. A layered approach is useful as well. If the source can be identified via netflow and null routed before it gets to the firewall and content layer, then all the better. This is much more tricky with DDOS so having robust firewall that can eat traffic is helpful. My 3 cents -b On Mon, Jan 4, 2010 at 7:35 PM, Christopher Morrow morrowc.li...@gmail.comwrote: On Mon, Jan 4, 2010 at 9:18 PM, jim deleskie deles...@gmail.com wrote: What Roland said, I've seen people do this, no rules in place, still was able to kill the box (firewall) with a single CPU server. not to pile on, but... +1 to roland here as well. I've seen more than enough folks put in a 'firewall' in front of their 'server' (say a mail server) and then watch that die long before the rest of the system did. Now, if you have equipment capable today of doing a few million session creates/second and you feel comfortable that you can keep track of how attacks grow vs your capacity stays the same and move ahead of the curve well enough, then... by all means do as you want :) There's a cost analysis which Roland sidestepped here as well, state-tracking at the rates required is expensive, as compared to relatively simple acls in hardware with no state on the upstream router. Spend where it matters, and make sure you understand where the failure points are that you place into your network. -chris -jim On Mon, Jan 4, 2010 at 10:04 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 4:25 AM, Jeffrey Lyon wrote: Use a robust firewall such as a Netscreen in front of your mitigation tool. Absolutely not - the firewall will fall over due to state-table exhaustion before the mitigation system will. Firewalls (which have no place in front of servers in the first place), load-balancers, and any other stateful devices should be southbound of the mitigation system. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken -- Bill Blackford Network Engineer
Re: D/DoS mitigation hardware/software needed.
With these safeguards in place - and with flow devices being part of the mix somewhere .. what you propose is quite reasonable. There's still the question of whether an application that receives a lot of new / untrusted traffic - a mail or web server - would benefit from having a stateful firewall in front .. Roland seems to think not. --srs On Tue, Jan 5, 2010 at 9:35 AM, Jeffrey Lyon jeffrey.l...@blacklotus.net wrote: 1. We have multiple nodes conducting DDoS scrubbing, one failing would not be catastrophic. 2. Indeed. 3. Sort of, such devices are downstream for extremely valid reasons I won't get into now. 4. Indeed, were equipped to handle substantially higher than 150kpps. I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I don't expect an employee of the vendor themselves to attest to this though. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote: I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I disagree with this proposition, too. S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed. I don't expect an employee of the vendor themselves to attest to this though. Wrong again. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 11:20 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 11:05 AM, Jeffrey Lyon wrote: I'm sure Arbor is really neat but I disagree that any DDoS appliance is a standalone solution. I disagree with this proposition, too. S/RTBH and/or flow-spec are great DDoS mitigation tools which don't require any further investment beyond the network infrastructure an operator has already purchased and deployed. These should be the first mitigation tools anyone deploys; in many cases, they're all that's needed. Is it fair to say that most folks in this thread believe there is not 'one size fits all', and that there are quite a few tools available in the security/networking toolbox? Some of these are outlined in past nanog tutorials: http://www.nanog.org/meetings/nanog23/presentations/greene.pdf http://www.nanog.org/meetings/nanog26/presentations/ispsecure.pdf http://www.nanog.org/meetings/nanog28/presentations/sink.pdf http://www.nanog.org/meetings/nanog36/presentations/greene.ppt Sorry to pick on barry here, but he's got a few preso's up from past NANOG's that cover this topic pretty well. All of them talk about a set of tools a network operator should be familiar with, with escalating costs (dollars and packet-loss/collateral damage), and some cut/pasteable configlets even. I don't expect an employee of the vendor themselves to attest to this though. Wrong again. eh, roland's always happy to bash on employers :) but, he's got some solid standing on this set of points. Again, if you know what you're doing then feel free to go off and do it, but at first blush there are LOTS of people putting 'servers' out on the 'public network' behind devices whoafully prepared to deal with 'real world traffic demands', so instead of making the same mistake, perhaps learning from some experience would be in order? The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short? (note I think Roland may have been party to some of the presenations I linked in this... I certainly was for one of them at least, in case that matters.) -Chris ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:41 AM, Christopher Morrow wrote: (note I think Roland may have been party to some of the presenations I linked in this... Yes, sir, and thanks for posting those links! You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention here have contributed greatly to advancing the state of the art and generating the body of real-world opsec material out there, which it behooves all network operators to study and take into consideration in their own contexts. I also highly recommend this book by Dave Smith and Gregg Schudel of Cisco - it's the best (and only!) book on real-world opsec out there, available in dead-tree, Kindle, and Adobe Reader formats: http://www.amazon.com/Router-Security-Strategies-Securing-Network/dp/1587053365/ref=sr_1_1?ie=UTF8s=booksqid=1262667257sr=8-1 [Full disclosure; I'm cited in the book, but received and continue to receive no renumeration of any kind due to same.] Here's another preso on this same topic which may be of interest: http://files.me.com/roland.dobbins/k54qkv --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:35 AM, Rick Ernst na...@shreddedmail.com wrote: I'm interested in seeing products (including software) that already have the development (anomaly detection, trends/reports, etc.) work done so I can spend my cycles elsewhere. This might fit the bill - http://www.zurich.ibm.com/aurora/ Now commercially available as http://www-01.ibm.com/software/tivoli/products/netcool-performance-flow/ Full disclosure - I work for big blue - but not in any division that works on Aurora / Tivoli Netcool. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote: A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Good plan. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Mon, Jan 4, 2010 at 9:08 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 12:05 PM, Rick Ernst wrote: A solution preferably that integrates with NetFlow and RTBH. An in-line solution obviously requires an appliance, or at least special/additional hardware. The key is to not be inline all the time, but only inline *when needed*. This removes operational complexity, provides the ability to oversubscribe, and simplifies the routine troubleshooting matrix. I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. I'm looking at taking the first whack at immediate mitigation at the border/edge (upstream) via uRPF and RTBH. Good plan. Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. I know you said generally, but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Note that my original question was in the context of a D/DoS composed of lots of itty-bitty packets. Other attack mechanisms do not necessarily lend themselves to chop 'em off at the knees. Rick
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland rdobb...@arbor.net wrote: Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. ... and manual wont scale in ddos -- Suresh Ramasubramanian (ops.li...@gmail.com)
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Christopher Morrow [mailto:morrowc.li...@gmail.com] Sent: Monday, January 04, 2010 11:41 PM The original poster seemed to be asking about appliance based solutions, so your pointed remarks about Roland aside he was actually answering the question. I wonder if Stefan Fouant would offer some of his experience with 'not arbor' vendor solutions to be used when other techniques come up short? Interesting thread! And I'm happy to chime in - thanks Chris! I too would have to strongly agree with Roland's comments about not front-ending your mitigation solution with firewalls or load-balancers - these are precisely the types of things which topple over first under a big attack, as such having your mitigation devices behind them makes little sense. If you must use firewalls and/or LBs, put them behind the mitigation device, where the traffic has already been scrubbed and your state tables won't be exhausted. Having said that, if you've got a router capable of doing generic packet filters upstream of your mitigation device, this is certainly a good place to apply stateless filters which can pitch any traffic you are sure you will never need to receive. Flowspec and/or automated blackhole routing works very well for this application when you want to get rid of certain types of cruft, before it hits your mitigation device. Now, on to the OPs original question regarding appliance based solutions, I would say I am actually a firm believer in having multiple vendors in place if you can afford it. Arbor definitely has a corner on the market here, with the most comprehensive solution which entails everything from detection to mitigation and pretty much everything in between. Arbor can also automate the FlowSpec process and/or generate router ACLs for propagation to upstream routers... They do a lot of other stuff as well such as identification of BGP hijacking, DNS analysis, can automate a lot of the RTBH or S/RTBH stuff. Where Arbor generally suffers is with sophisticated attack traffic which requires complex mitigations - these often require a lot of tweaking in order to properly scrub the traffic... knowing your environment and which attack vectors are likely to be exploited is your best bet here, where you can configure mitigation templates in advance for rapid deployment during an attack. I've also worked with the RioRey devices and I have to say that although they don't have all the bells and whistles that some of the other vendors offer, their approach to mitigation is entirely unique and can genuinely mitigate attacks in record-time. Without disclosing too much of their intellectual property, I will say that their algorithms essentially look at the randomness and probability of address space distribution within the attack traffic, and can generally offer a high degree of certainty of scrubbing the majority of the bad traffic - and they do this WITHOUT having to do DPI as many other vendors are currently doing. Bottom line - if you're looking for something with a lot of bells and whistles and the ability to monitor/detect/analyze/etc., you're probably better off going with an Arbor solution. If you have lower OpEx and just want something that you can set it and forget it, you'd be hard pressed not to look at the RioRey. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:19 PM, Suresh Ramasubramanian wrote: ... and manual wont scale in ddos Actually, it can and does. ; I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:19 PM, Rick Ernst wrote: I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. I strongly disagree with this, except for properties which are under sustained attack 24/7. If one has constructed one's detection/classification/traceback/mitigation system properly, one always knows at a glance the state of the system. Otherwise, whenever there's any issue whatsoever with the properties under protection, one must try and prove a negative - i.e., that the mitigation solution isn't causing the problem. Happens every time, heh. I know you said generally, but if I'm seeing 200Kpps from a.b.c.d, I don't care if a.b.c.d is spoofed. I want the traffic blocked from the guts of my network. Not if it's legit, you don't, or if the attacker is spoofing, say, the IPs of the root nameservers, or the TLDs, or an e-commerce/supply-chain partner . . . or if the attack is originating behind a broadband mega-proxy, or a mobile CGN. ; Also, if you've a variety of tools at your disposal, like S/RTBH and/or flow-spec, and then more sophisticated (and expensive) tools like IDMS, the freedom to choose the least intrusive/most situationally-appropriate tool to mitigate a given attack is essential for resource preservation and the ability to oversubscribe the more sophisticated tools. Note that my original question was in the context of a D/DoS composed of lots of itty-bitty packets. Other attack mechanisms do not necessarily lend themselves to chop 'em off at the knees. Absolutely, which is where the situationally-specific selection of tools/modes comes into play. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 5, 2010 at 10:52 AM, Dobbins, Roland rdobb...@arbor.net wrote: I'm referring to the employment and selection of situationally-appropriate tools, mind. The tools themselves must of necessity perform their work in a largely automated fashion once they're employed, which is what I believe you actually meant. fair enough. -- Suresh Ramasubramanian (ops.li...@gmail.com)
Re: D/DoS mitigation hardware/software needed.
On Tue, Jan 05, 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Has anyone deployed a DDoS distributed enough to inject ETOOMANY routes into the hardware forwarding tables of routers? I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective? Adrian
Re: D/DoS mitigation hardware/software needed.
I think you, Roland, and I are all agreeing on the same argument. The (my) confusion entered with the statement of, The key is to not be inline all the time, but only inline *when needed*. I inferred a topological or physical path change from that. Redirecting traffic (which is really just an extension of RTBH; a scrubber destination rather than Null0) is an understandable state. Rick On Mon, Jan 4, 2010 at 9:34 PM, Stefan Fouant sfou...@shortestpathfirst.net wrote: -Original Message- From: Rick Ernst [mailto:na...@shreddedmail.com] Sent: Tuesday, January 05, 2010 12:19 AM I'd argue just the opposite. If your monitoring/mitigation system changes dependent on the situation (normal vs under attack), you are adding complexity to the system. What mode is the system in right now? Is this customer having connectivity issues because of a state change in the network? etc. Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Suresh Ramasubramanian [mailto:ops.li...@gmail.com] Sent: Tuesday, January 05, 2010 12:19 AM On Tue, Jan 5, 2010 at 10:38 AM, Dobbins, Roland rdobb...@arbor.net wrote: Additional mitigation would be via manual or automatic RTBH or security/abuse@ involvement with upstreams. Automagic is generally bad, as it can be gamed. ... and manual wont scale in ddos There are pros and cons to each approach. Certain types of things can be automated, in fact I've done this using the Auto-mitigate feature in Arbor coupled with pre-configured mitigation templates for certain types of traffic and it works very well. But generally, as Roland mentioned automagic stuff can be gamed and for the majority of the stuff you are going to want an operator to look at the alert before making the decision to offramp. The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the big red button to offramp and enable the mitigation. Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Adrian Chadd wrote: I mean, I assume that there's checks and balances in place to limit then number of routes being injected into the network so one doesn't overload the tables, but what's the behaviour if/when this limit is reached? Does mitigation cease being as effective? For IDMS 'scrubbing' solutions, one merely injects the route of the attack targets into one's iBGP, in order to draw all traffic towards that specific target into the scrubbing center. For S/RTBH and flow-spec, modern edge routers can scale to millions of routes; also note one isn't limited to /32s. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Stefan Fouant wrote: The trick is to try to automate as much around the process as possible - I've worked in environments where just making little changes to incident handling response methods reduced the time to mitigate an attack from hours to minutes, all the while still requiring an operator to press the big red button to offramp and enable the mitigation. Concur 100% - and when the end-customer is under attack and screaming, this reduction in time to detect/classify/traceback/mitigate makes all the difference. Your very salient comments highlight the paramount importance of preparation as the key enabling phase of the six-phase security incident-handling methodology: 1. Preparation. 2. Detection/identification. 3. Classification. 4. Traceback. 5. Reaction. 6. Post-mortem (feeding lessons learned back into the Preparation phase). --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 12:39 PM, Rick Ernst wrote: I think you, Roland, and I are all agreeing on the same argument. GMTA. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 11:57 AM, Dobbins, Roland wrote: You and Barry and Tim Battles and Sean Donelan and Danny McPherson and Don Smith and Steve Bellovin and Jared Mauch and John Kristoff and a lot of other folks too numerous to mention . . . including Paul Vixie, Darrel Lewis, Ryan McDowell, Paul Quinn, Michael Behringer, Craig Huegen, Craig Labvoitz, Dave Smith, Gregg Schudel, and many others. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
RE: D/DoS mitigation hardware/software needed.
On Tue, 5 Jan 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? -Hank
RE: D/DoS mitigation hardware/software needed.
-Original Message- From: Hank Nussbacher [mailto:h...@efes.iucc.ac.il] Sent: Tuesday, January 05, 2010 1:02 AM On Tue, 5 Jan 2010, Stefan Fouant wrote: Almost all of the scalable DDoS mitigation architectures deployed in carriers or other large enterprises employ the use of an offramp method. These devices perform a lot better when you can forward just the subset of the traffic through as opposed to all. It just a simple matter of using static routing / RTBH techniques / etc. to automate the offramp. That said, what are all those ISPs doing now that Cisco has stopped developing the Guard? Well of course, moving to Arbor haha... eased in part by a little Cisco initiative called Clean Pipes 2.0 :) Stefan Fouant, CISSP, JNCIE-M/T www.shortestpathfirst.net GPG Key ID: 0xB5E3803D
Re: D/DoS mitigation hardware/software needed.
I actually agree with most of this, but want to correct a clearly inadvertent error below, and make a couple of points. PCI DSS does not require a Web application firewall. It requires that OR an independent audit of all code within PCI scope by a third party. If a WAF is used, it only need be deployed in such a manner as to protect devices inside of PCI scope (depending on design, this may or may not include everything that has public exposure). The powers that be specified additional methods by which PCI compliance could be satisfied other than just these two after the wailing of the masses. I don't know off the top of my head if those other methods will be acceptable in 2010. Personally, I believe a DOS detection/prevention system is typically going to be best placed between screening routers/switches and the next L3/L4 aware device- generally you will want it (or them) to be as close to your ingress edge as you can put it- why allow DOS traffic to go further downstresm? So I suspect Roland and I agree on that fwiw. Earlier, Roland mentions load-balancers and firewalls as both being susceptible to state-table overflows. Certainly this is possible and happened in the past, and it argues for a DOS prevention device being in front of them. At least one modern high-end load balancer handles this well and is in front of a large number if not the majority of major sites. It is possible to build equipment that can handle vast numbers of state entries and handle lookups into them in very attractive big O's. It is also possible to build systems that gracefully handle table exhaustion and enter into aggressive reaping modes. This has been empirically proven to me wrt load-balancers. Some device is going to have to handle the state and insert itself into the path- even if that is a lone webserver. I maintain that there is no reason to believe that it is not possible for a firewall to do that as well as a load-balancer. As for whether you should have a stateful firewall in front of a production web server farm, I understand Roland's point. However, I will say that there are many reasons why people put firewalls in front of webservers- to name some: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? * Location for IDS/IDP. * Connection cleanup, re-assembling fragments, etc. * SYN flood protection, etc. * Single choke point to block incoming traffic deemed undesirable. * Single log point for inbound connections for analysis and auditing requirements. * Allows outbound traffic enforcement. * Allows conditional inbound traffic from specific approved external hosts- e.g. a partner. * Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces. * In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better. * Yeah, auditors like it. I'm not at all adverse to putting transparent proxies in front of your website. CDN vendors aren't either. I will say that I have had several bad experiences with WCCP not working as expected and failing non-gracefully. I am not saying its always a good idea to have a stateful firewall in front of your web servers, I'm saying that there are reasons why you might want to and in my experience it is common. --D On Mon, Jan 4, 2010 at 7:31 PM, Dobbins, Roland rdobb...@arbor.net wrote: On Jan 5, 2010, at 10:18 AM, Suresh Ramasubramanian wrote: 5 Ditch the stateful firewall and exclusively use a netflow device NetFlow analysis is very useful for network visibility, and detection/classification/traceback. There are both open-source and commercial NetFlow collection and analysis systems available (full disclosure: I work for a vendor of both NetFlow analysis and DDoS mitigation solutions); however, they don't provide mitigation, which is where S/RTBH, flow-spec, and/or IDMS come into play. PCI DSS iatrogenically *requires* that a 'Web application firewall' be placed in front of Web servers which process credit card information (PCI DSS completely ignores availability, and contains a number of recommendations which are actually harmful from an opsec standpoint). Running mod_security or its equivalent on the front-end Web servers themselves fulfills this requirement without putting a stateful DDoS chokepoint in front of the servers. It's also a really good idea to front Web servers with a tier of caching-only transparent reverse proxies; Squid is a good choice for this, as well as various commercial offerings. WCCPv2 clustering (supported by Squid and several commercial caching/proxying solutions) allows this tier to be scaled horizontally in order to meet capacity demands.
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: * Defense in depth. You've never had a host that received external traffic ever accidentally have iptables or windows firewall turned off? Even when debugging a production outage or on accident? Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. 'Stateful inspection' where in fact there is no useful state to inspect is pointless. * Location for IDS/IDP. Non-sequitur, as these things have nothing to do with one another (plus, these devices are useless, anyways, heh). * Connection cleanup, re-assembling fragments, etc. Far, far, far better and more scalably handled by the hosts themselves and/or load-balancers. * SYN flood protection, etc. Firewalls simply don't handle this well, marketing claims aside. They crash and burn. * Single choke point to block incoming traffic deemed undesirable. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Single log point for inbound connections for analysis and auditing requirements. Contextless, arbitrary syslog from firewalls and other such devices is largely useless for this purpose. NetFlow combined with server/app/service logs is the answer to this requirement. * Allows outbound traffic enforcement. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Allows conditional inbound traffic from specific approved external hosts- e.g. a partner. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Some firewalls allow programmatic modification of configurations with all the benefits/pain that brings. This is alongside traditional CLI and GUI interfaces. Ugly, brittle, siloed, to be avoided at all costs. * In some/many cases a zone based firewall configuration can be much easier to work with than a large iptables config. Certainly the management tools are better. Again, policy should be enforced via stateless ACLs in router/switch hardware capable of handling mpps. * Yeah, auditors like it. Education is the answer here. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
Re: D/DoS mitigation hardware/software needed.
On Jan 5, 2010, at 2:38 PM, Darren Bolding wrote: PCI DSS does not require a Web application firewall. http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1313797,00.html Since no business is going to allow an external 'code review' (if it's even possible, given that they're likely using COTS products, the source code of which they simply don't have), this defaults to a requirement for the 'Web application firewall'. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Injustice is relatively easy to bear; what stings is justice. -- H.L. Mencken
