ddos attack blog
Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help. For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there... Regards, CB
Re: ddos attack blog
On Feb 13, 2014, at 12:06 PM, Cb B wrote: > Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, > DTAG and others > > http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack > > Standard plug for http://openntpproject.org/ and > http://openresolverproject.org/ and bcp38 , please fix/help. > > For those of you paying attention to the outage list, this is a pretty > big deal that has had daily ramification for some very big networks > https://puck.nether.net/pipermail/outages/2014-February/date.html > > In general, i think UDP is doomed to be blocked and rate limited -- > tragedy of the commons. But, it would be nice if folks would just fix > the root of the issue so the rest of us don't have go there... While I'm behind some of the inventory projects (so you can go ahead and fix.. let me know if you need/want the URLs to see data for your networks)... I must provide credit to those behind the "Amplification Hell" talk at NDSS. If you are at all interested in what is going on, you should attend or review the content. http://www.internetsociety.org/ndss2014/programme BCP-38 on your customers is going to be critical to prevent the abuse reaching your network. Please ask your vendors for it, and ask for your providers to filter your network to prevent you originating this abuse. If you operate hosted VMs, servers, etc.. please make sure those netblocks are secured as well. You can easily check your network (As can the bad guys!) here: http://spoofer.cmand.org/ - Jared
Re: ddos attack blog
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/13/2014 9:06 AM, Cb B wrote: > Good write up, includes name and shame for AT&T Wireless, IIJ, > OVH, DTAG and others > > http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack > > Standard plug for http://openntpproject.org/ and > http://openresolverproject.org/ and bcp38 , please fix/help. > > For those of you paying attention to the outage list, this is a > pretty big deal that has had daily ramification for some very big > networks > https://puck.nether.net/pipermail/outages/2014-February/date.html > > In general, i think UDP is doomed to be blocked and rate limited > -- tragedy of the commons. But, it would be nice if folks would > just fix the root of the issue so the rest of us don't have go > there... > The alternative is get people to understand that anti-spoofing is good, and efforts to combat spoofing should be encouraged. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL9AR4ACgkQKJasdVTchbJZYwEAivI00Yq7RSMze74GFQKEyCeH pS2s8TH0ba08NWKC22AA/jyN35xonJBzldJA8/xlzhnuLnyOFB0Y7GKZ8NiqRiRl =ItxR -END PGP SIGNATURE-
Re: ddos attack blog
On 02/13/2014 10:06 AM, Cb B wrote: Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help. For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there... UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices. Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol. --John Regards, CB
Re: ddos attack blog
On Feb 13, 2014, at 1:47 PM, John wrote: > On 02/13/2014 10:06 AM, Cb B wrote: >> Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, >> DTAG and others >> >> http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack >> >> Standard plug for http://openntpproject.org/ and >> http://openresolverproject.org/ and bcp38 , please fix/help. >> >> For those of you paying attention to the outage list, this is a pretty >> big deal that has had daily ramification for some very big networks >> https://puck.nether.net/pipermail/outages/2014-February/date.html >> >> In general, i think UDP is doomed to be blocked and rate limited -- >> tragedy of the commons. But, it would be nice if folks would just fix >> the root of the issue so the rest of us don't have go there... > > UDP won't be blocked. There are some vendors that have their own hidden > protocol inside UDP packets to control and communicate with their devices. > > Thinking on it again, maybe blocking UDP isn't all that bad. Would force the > vendors to not 'hide' their protocol. > Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment. I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks. I've seen maybe 100-200 per-ASN reports handed out to network operators. If you want yours, please e-mail ntp-s...@puck.nether.net to obtain it. Put your ASN in the subject line and/or body. - Jared (and others like Patrick that presented on the projects behalf).
Re: ddos attack blog
On Friday, February 14, 2014 03:01:27 AM Jared Mauch wrote: > I would actually like to ask for those folks to un-block > NTP so there is proper data on the number of hosts for > those researching this. The right thing to do is > reconfigure them. I've seen a good trend line in NTP > servers being fixed, and hope we will see more of that > in the next few weeks. Depending on your OS, the fixes can be quite simple or interesting. On my FreeBSD servers, simply updating with "freebsd-update" was enough to fix the issue (in addition to limiting who/what can access the service). On Cisco devices, the ACL's you can attach to the NTP process are quite effective. On Juniper devices, it is less intuitive, and even though NTP is enabled only as a client, it, sadly, runs the server as well. A firewall filter helps here when applied correctly. Can't speak to other OS's. Mark. signature.asc Description: This is a digitally signed message part.
Re: ddos attack blog
On Thu, Feb 13, 2014 at 08:01:27PM -0500, Jared Mauch wrote: > I would actually like to ask for those folks to un-block NTP so there is > proper data on the number of hosts for those researching this. The right > thing to do is reconfigure them. I've seen a good trend line in NTP servers > being fixed, and hope we will see more of that in the next few weeks. A slight exception to that statement, if I may... The right thing to do is for people to not permit services to operate on hosts they do not intend to operate on and not to be visible to those they do not intend to use them. In other words, to properly manage their networks. If that means blocking all access to potentially faulty implementations, then that's the right thing to do. In short, companies should do what is right for their companies and nevermind anyone else. Never forget that researches are just part of the "public" and should never consider that their usage of the internet is any more or less valid to the average third party than the next guy. -Wayne --- Wayne Bouchard w...@typo.org Network Dude http://www.typo.org/~web/
Re: ddos attack blog
On 02/13/2014 06:01 PM, Jared Mauch wrote: On Feb 13, 2014, at 1:47 PM, John wrote: UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices. Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol. Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment. I was being a bit extreme, I don't expect UDP to be blocked and there are valid uses for NTP and it needs to pass. Can you imagine the trading servers not having access to NTP? The knee jerk reaction to just block NTP is a temporary measure that can be used while other mitigation steps are implemented. I kinda hijacked the NTP issue a bit and expanded it to cover the undocumented uses of device control in UDP. I'll leave that issue for another day, just wanted to raise awareness if it was not already known. --John I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks. I've seen maybe 100-200 per-ASN reports handed out to network operators. If you want yours, please e-mail ntp-s...@puck.nether.net to obtain it. Put your ASN in the subject line and/or body. - Jared (and others like Patrick that presented on the projects behalf).
Re: ddos attack blog
> I was being a bit extreme, I don't expect UDP to be blocked and there are > valid uses for NTP and it needs to pass. Can you imagine the trading > servers not having access to NTP? Sure. They could setup internal NTP servers listening to GPS. Would it be as good overall as using external servers? Probably not, but it might be good enough. I doubt if it would be very high on any trading floors list of nasty problems. They could arrange to poke holes through the generic UDP block - whitelist the few known cases where UDP traffic is expected. Would it be a pain to administer? Probably, but I'll bet it could be made to work. -- These are my opinions. I hate spam.
Re: ddos attack blog
On 2/14/14, 3:00 PM, Hal Murray wrote: > >> I was being a bit extreme, I don't expect UDP to be blocked and there are >> valid uses for NTP and it needs to pass. Can you imagine the trading >> servers not having access to NTP? > > Sure. > > They could setup internal NTP servers listening to GPS. Would it be as good > overall as using external servers? Probably not, but it might be good > enough. I doubt if it would be very high on any trading floors list of nasty > problems. > > They could arrange to poke holes through the generic UDP block - whitelist > the few known cases where UDP traffic is expected. Would it be a pain to > administer? Probably, but I'll bet it could be made to work. High value concentrated applications are relatively easy things to get high quality time to. it's all the consumer electronics devices and everything that uses ssl/tls that needs access to time that is a more diffuse and less tractable problem. joel > signature.asc Description: OpenPGP digital signature
Permitting spoofed traffic [Was: Re: ddos attack blog]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/14/2014 10:22 AM, Wayne E Bouchard wrote: > On Thu, Feb 13, 2014 at 08:01:27PM -0500, Jared Mauch wrote: >> I would actually like to ask for those folks to un-block NTP so >> there is proper data on the number of hosts for those researching >> this. The right thing to do is reconfigure them. I've seen a >> good trend line in NTP servers being fixed, and hope we will see >> more of that in the next few weeks. > > > A slight exception to that statement, if I may... > > The right thing to do is for people to not permit services to > operate on hosts they do not intend to operate on and not to be > visible to those they do not intend to use them. In other words, to > properly manage their networks. If that means blocking all access > to potentially faulty implementations, then that's the right thing > to do. In short, companies should do what is right for their > companies and nevermind anyone else. > > Never forget that researches are just part of the "public" and > should never consider that their usage of the internet is any more > or less valid to the average third party than the next guy. > Taken to the logical extreme, the "right thing" to do is to deny any spoofed traffic from abusing these services altogether. NTP is not the only one; there is also SNMP, DNS, etc. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL+Y68ACgkQKJasdVTchbJ/dgEAqgERvP6HMl2v5fbhZDwI9QKT YEe/c3mN5gZlxsIKFo0A/3BH9KMV6ln7XMrlnk4c/GuwZ9X4LAgqO6l2p8u3aA49 =yWZU -END PGP SIGNATURE-
Re: Permitting spoofed traffic [Was: Re: ddos attack blog]
On 2/14/2014 12:42 PM, Paul Ferguson wrote: Taken to the logical extreme, the "right thing" to do is to deny any spoofed traffic from abusing these services altogether. Since the 1990s I have argued (ineffectively, it turns out) a case that says that sentence can be edited down to good advantage as: > Taken to the logical extreme, the "right thing" to do is to deny any > spoofed traffic. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)
Re: Permitting spoofed traffic [Was: Re: ddos attack blog]
On Fri, Feb 14, 2014 at 10:42:55AM -0800, Paul Ferguson wrote: [snip] > Taken to the logical extreme, the "right thing" to do is to deny any > spoofed traffic from abusing these services altogether. NTP is not the > only one; there is also SNMP, DNS, etc. ...and then we're back to "implement BCP38 already!" (like one of the authors of the document didn't think of that, ferg? ;-) NB: Some Entities believe all filtering is 'bcp 38' and thus have given this stone-dead logical and sane practice a bad rap. If someone is sloppy with their IRR-based filters or can't drive loose RPF correctly, that isn't the fault of BCP38. The document specifically speaks to aggregation points, most clearly in the introduction: "In other words, if an ISP is aggregating routing announcements for multiple downstream networks, strict traffic filtering should be used to prohibit traffic which claims to have originated from outside of these aggregated announcements." This goes for access, hosting, and most recently virtual hosting in teh cloude. Stop forgery at your edges and your life will be easier. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / CotSG / Usenix / NANOG
Re: Permitting spoofed traffic [Was: Re: ddos attack blog]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/14/2014 3:00 PM, Larry Sheldon wrote: > On 2/14/2014 12:42 PM, Paul Ferguson wrote: >> Taken to the logical extreme, the "right thing" to do is to deny >> any spoofed traffic from abusing these services altogether. > > Since the 1990s I have argued (ineffectively, it turns out) a case > that says that sentence can be edited down to good advantage as: > >> Taken to the logical extreme, the "right thing" to do is to deny >> any spoofed traffic. > But of course. :-) - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL+y1QACgkQKJasdVTchbIgWgEAns/Nw6pqK+BaXSmI2DmP5J9Z mxeVg3KTCHdMBSDeZXAA/2+PlVSwHXdFem6hwRC/iv1+zscghkwUgimGbhKA5Gnx =VXx2 -END PGP SIGNATURE-
Re: Permitting spoofed traffic [Was: Re: ddos attack blog]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/14/2014 4:09 PM, Joe Provo wrote: > On Fri, Feb 14, 2014 at 10:42:55AM -0800, Paul Ferguson wrote: > [snip] >> Taken to the logical extreme, the "right thing" to do is to deny >> any spoofed traffic from abusing these services altogether. NTP >> is not the only one; there is also SNMP, DNS, etc. > > ...and then we're back to "implement BCP38 already!" (like one of > the authors of the document didn't think of that, ferg? ;-) > > NB: Some Entities believe all filtering is 'bcp 38' and thus have > given this stone-dead logical and sane practice a bad rap. If > someone is sloppy with their IRR-based filters or can't drive loose > RPF correctly, that isn't the fault of BCP38. > > The document specifically speaks to aggregation points, most > clearly in the introduction: "In other words, if an ISP is > aggregating routing announcements for multiple downstream networks, > strict traffic filtering should be used to prohibit traffic which > claims to have originated from outside of these aggregated > announcements." > > This goes for access, hosting, and most recently virtual hosting in > teh cloude. Stop forgery at your edges and your life will be > easier. > Indeed -- I'm not in the business of bit-shipping these days, so I can't endorse or advocate any particular method of blocking spoofed IP packets in your gear. I can, however, say with confidence that it is still a good idea. Great idea, even. :-) - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL+y8sACgkQKJasdVTchbKTXAEA0/czP0ECsFX4CyUr6yt4Dkap D0NZT/UIo6h5E/dl0KEA/3hpxN2NLxZRix6JUTVHyv+LZ4RzgpG2myoXbgAq1+WS =QQjA -END PGP SIGNATURE-
Re: Permitting spoofed traffic [Was: Re: ddos attack blog]
On 2/14/2014 9:07 PM, Paul Ferguson wrote: > Indeed -- I'm not in the business of bit-shipping these days, so I > can't endorse or advocate any particular method of blocking spoofed IP > packets in your gear. If you're dead-end, a basic ACL that permits ONLY your prefixes on egress, and blocks your prefixes on ingress, is perhaps the safest bet. Strict uRPF has it's complications, and loose uRPF is almost too forgiving. If you're providing transit, it gets much more complicated much more quickly, but the same principles apply (they just get to be a less-than-100% solution) :) > I can, however, say with confidence that it is still a good idea. > Great idea, even. :-) Oh yeah :) Jeff signature.asc Description: OpenPGP digital signature
