Re: [uknof] DNS/NTP censured, a solution !
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 13/02/2014 18:04, Robin Williams wrote: It'd also be good to discuss merging data from these projects into an upstream 'open-generalbadstuff-project'. shadowserver are putting some source of open dns resolvers into their reports and I'm sure that if we asked nicely they could start using sources of ntp data too. www.cisp.org.uk can offer you a feed of generalbadstuff-seen-on-your-asn through a package called AbuseHelper (that we'll also been deploying shortly). I believe it includes the shadowserver data. We'll be using it to replace our ad-hoc scripts for dealing with all these separate sources. James - -- James Davis0300 999 2340 (+44 1235 822340) Senior CSIRT Member Lumen House, Library Avenue, Didcot, Oxfordshire, OX11 0SG -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS/fUJAAoJEHRLPxE0xhCCwRgIAJHQYWW7rLMKJNVe1yKcblQC K5/s0jzmtrPB934QKSoEXpTEWnLDwvHekM7fzEsfqtnEHBJQV58mzMK4W/URczQ6 890rLCtD52PLLxYPaZub7IgsFOyxaYwk8Ej6hI7xooPnD8yvm/HjEyZg0xG5JCMU 8RTDJQ6zn4AwZzRE+tDhgyhO1pDt4zH+AkgUYG5s+3EoXdDwJmCRKcq8QjPkoBhY +DoAk6oN87iQkQypBOduQxSRhPfViPiVkMnRG8hfv+0LS9S5NMBhlR+eyBR+4uNy lZIzO7lU937EGwdQrr7OsWnYEpu4DFvIaXa5BBTXmX/ixEdNeG9ZXw1l6Txt0Mo= =5GMy -END PGP SIGNATURE- Janet(UK) is a trading name of Jisc Collections and Janet Limited, a not-for-profit company which is registered in England under No. 2881024 and whose Registered Office is at Lumen House, Library Avenue, Harwell Oxford, Didcot, Oxfordshire. OX11 0SG. VAT No. 614944238
Re: [uknof] DNS/NTP censured, a solution !
Keith Mitchell wrote: But it's not just about NTP and DNS, pretty much any UDP-based service that can do amplification is in play, e.g SNMP, Chargen and I've even seen QOTD (UDP 19). snip Universal BCP38 source address validation is needed more badly then ever :-( It really is. Glad to hear it's not just us - reflection attacks are starting to be one of the biggest problems we have - and it's proving incredibly difficult to deal with. It seems that any script kiddie that wants to can launch attacks of overwhelming volume - and boy do they do so. :( Pretty much every single day we're seeing multiple 25G+ attacks now, we've had a fair few in the 40-80G and a reported 100G+ last week too - usually directed at single machines between both our own network as well as the 'off-net' carriers we use around the world. It's pretty much always NTP or DNS reflection - but we've seen loads of chargen, echo, daytime, SNMP and random fragmented packets too. We can ACL most of it out our side and protect the target, but providing many-tens-of-gigabits worth of capacity just to soak up attack traffic is 'difficult' to sustain financially! That leaves blackholing via transit providers (not peering though) - but doesn't really solve the problem. I don't know what the 'end result' of this is going to be - but i'm sure that even if the NTP / DNS amplifiers get cleaned up enough to fix that, there's no shortage of other potential amplifiers out there anyway. If BCP38 doesn't start to gain wider adoption, this is just going to keep getting worse.
Re: [uknof] DNS/NTP censured, a solution !
On 14/02/2014 11:54, Giles Davis wrote: Keith Mitchell wrote: Universal BCP38 source address validation is needed more badly then ever :-( It really is. It really is, but bear in mind that a single 1GE connection with no urpf can be used to create ~250-300G of backscatter traffic. This means that there's only a requirement to have a single unscrupulous or incompetent ISP with GE in the world to allow a devastating DoS to be launched against anyone anywhere. Nick
Re: [uknof] DNS/NTP censured, a solution !
Nick Hilliard wrote: It really is, but bear in mind that a single 1GE connection with no urpf can be used to create ~250-300G of backscatter traffic. This means that there's only a requirement to have a single unscrupulous or incompetent ISP with GE in the world to allow a devastating DoS to be launched against anyone anywhere. Indeed - which is certainly a problem! :) So what's the 'proper' solution to all this then beyond just adding enough capacity to absorb ever larger attacks? How's this going to end up? There must be plenty of businesses who this kind of thing is seriously affecting - and the trend upwards in size of attacks has been absolutely massive over the past year so it doesn't take long to hit a point where adding bandwidth just isn't affordable. When pretty much anyone who wants to can just knock you offline and there's very little you can do about it, something is going to have to happen. At this point we've not seen any threats or demands as a result of these attacks - as far as we know it's just kids doing it 'cos they can' - but there doesn't seem to be a solution in sight either beyond 'turn the target(s) off until they stop'.
Re: [uknof] DNS/NTP censured, a solution !
On 02/14/2014 06:54 AM, Giles Davis wrote: Keith Mitchell wrote: But it's not just about NTP and DNS, pretty much any UDP-based service that can do amplification is in play, e.g SNMP, Chargen and I've even seen QOTD (UDP 17). Universal BCP38 source address validation is needed more badly then ever :-( I don't know what the 'end result' of this is going to be - but i'm sure that even if the NTP / DNS amplifiers get cleaned up enough to fix that, there's no shortage of other potential amplifiers out there anyway. If BCP38 doesn't start to gain wider adoption, this is just going to keep getting worse. For one perspective: http://queue.acm.org/detail.cfm?id=2578510 Keith
Re: [uknof] DNS/NTP censured, a solution !
On 02/14/2014 09:02 AM, Giles Davis wrote: Nick Hilliard wrote: It really is, but bear in mind that a single 1GE connection with no urpf can be used to create ~250-300G of backscatter traffic. This means that there's only a requirement to have a single unscrupulous or incompetent ISP with GE in the world to allow a devastating DoS to be launched against anyone anywhere. Indeed - which is certainly a problem! :) So what's the 'proper' solution to all this then beyond just adding enough capacity to absorb ever larger attacks? How's this going to end up? What's happening now is that reactive, specific measures are being taken - protocol-specific vulnerabilities (e.g. RRL for DNS, disable monlist for NTP) are being plugged, ISPs are deploying better instrumentation to detect attack flows, and are turning on uRPF/other source-address filtering towards the worst traffic sources. The problem with these approaches is that: - they are just going to lead to an endless game of whack-a-mole as the bad guys find ever more reflection vectors which need plugging - this arms race will in turn educate the bad guys to be smarter - the vendors of security products are going to be more interested in selling bigger faster $olutions than tackling the underlying problems (cf ever-increasing claims for how big an attack various vendors claim to have dealt with) - TPTB are more likely to blame the Internet industry and take regulatory measures against us as an easy target than tackle the actual bad guys Something I think that would make a bigger difference would be for data to be gathered and published that names-and-shames those providers that don't do BCP38 source address validation. As an industry we then need to start contractually enforcing, de-peering and blocking traffic to/from those providers who don't take action to remedy this. The other thing is to beat up on our vendors - I hear many stories of how BCP38 cannot be implemented by people who want to, due to some bug or missing feature with CPE/edge/aggregation/core equipment. If self-regulation doesn't work, we can expect regulation. While mandating SAV/BCP38 would IMHO be a much more useful single item of legislation to reduce Internet evil than the swathes of vested-interest pandering nonsense we've had from our governments and regulators lately, it's hard to trust them to do it right. In any case, there are also many (probably most) nation-states out there proudly declaring that they have cyberwarfare capability, and it's hard to see how this is credible without a DDoS element. It might actually take international Internet disarmament treaties to nail this problem :-( Keith
Re: [uknof] DNS/NTP censured, a solution !
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/02/2014 23:07, Robin Williams wrote: Interesting timing - we've also been seeing a big increase in the same over the last few weeks, mainly targeting schools from automated ( cheap!) online 'booter' services (presumably instigated by students who have had enough of their IT lessons). If you are seeing attacks against schools and Janet is upstream - please let us know as and when it occurs. Even if all that we do (and we try to do a lot more) is add it to our statistics it's still valuable to build up a picture of activity for the rest of the community. We can be contacted at i...@csirt.ja.net or 0300 999 2340. Thanks, James - -- James Davis0300 999 2340 (+44 1235 822340) Senior CSIRT Member Lumen House, Library Avenue, Didcot, Oxfordshire, OX11 0SG -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS/NTNAAoJEHRLPxE0xhCCYQYIAM18kudXagMO4PkA4U66DeFW anZWm0BAwBPM2ap1dykbWOXdN1MztKehu3fLi7iv2sqB2tuC47smoi9fwJsuPvkM KWMFmleDGSBMvzDqP1sCKl7/FtyGyoQ/3Y5XwO2GZP0JC6v9CJNhxjiL8qdN/gly jxlRFzKc8rKnaOZHXx3KsL9515FTkc3AJfN9B/Aiaa7KWXeNVvTQ/pQ1tZYaNyVX +FcATQw1ig2y/RqsSeMRa3PVbTVwZ5H9Er9BPHyV/yg6waBoADU1qIlDjxFDvAhh C2eu4ACgl4ImgyjagWQALEDAY4RLh1loeKO76Wx++812tuyQNBKgxQ6hqv0eSaM= =ApMg -END PGP SIGNATURE- Janet(UK) is a trading name of Jisc Collections and Janet Limited, a not-for-profit company which is registered in England under No. 2881024 and whose Registered Office is at Lumen House, Library Avenue, Harwell Oxford, Didcot, Oxfordshire. OX11 0SG. VAT No. 614944238
Re: [uknof] DNS/NTP censured, a solution !
On 13/02/14 17:14, Keith Mitchell wrote: On 02/12/2014 06:37 PM, Wright, Matthew wrote: List of open NTP servers from http://openntpproject.org/ Also http://www.openresolverproject.org But it's not just about NTP and DNS, pretty much any UDP-based service that can do amplification is in play, e.g SNMP, Chargen and I've even seen QOTD (UDP 19). Yep, one that hit us the other week was UDP Chargen. After seeing the source port in flows, I tried a few of them on TCP 19 as well, and to my surprise, there it was. And there was me thinking Chargen was a thing of the 80's! It'd be nice to be able to automatically pull the full lists from these various scanning projects to use in statistical analysis as part of DDoS mitigation (i.e. if my traffic has just shot up and the majority of it is coming from IPs listed in these databases, I can take a pretty fair bet at what's happening and start to rate limit or temporarily block these sources). Anyone know if there is an interface for automated downloading of the raw data? Is anyone involved in these projects on list? It looks like you can request the data manually at the moment. It'd also be good to discuss merging data from these projects into an upstream 'open-generalbadstuff-project'. Cheers, Robin
Re: [uknof] DNS/NTP censured, a solution !
It would also be useful to be able to run resolver scans via ASN or larger block reports too. Limited to a /22 takes a fair old while. Peter Knapp -Original Message- From: uknof [mailto:uknof-boun...@lists.uknof.org.uk] On Behalf Of Robin Williams Sent: 13 February 2014 18:05 To: Keith Mitchell Cc: uknof@lists.uknof.org.uk Subject: Re: [uknof] DNS/NTP censured, a solution ! On 13/02/14 17:14, Keith Mitchell wrote: On 02/12/2014 06:37 PM, Wright, Matthew wrote: List of open NTP servers from http://openntpproject.org/ Also http://www.openresolverproject.org But it's not just about NTP and DNS, pretty much any UDP-based service that can do amplification is in play, e.g SNMP, Chargen and I've even seen QOTD (UDP 19). Yep, one that hit us the other week was UDP Chargen. After seeing the source port in flows, I tried a few of them on TCP 19 as well, and to my surprise, there it was. And there was me thinking Chargen was a thing of the 80's! It'd be nice to be able to automatically pull the full lists from these various scanning projects to use in statistical analysis as part of DDoS mitigation (i.e. if my traffic has just shot up and the majority of it is coming from IPs listed in these databases, I can take a pretty fair bet at what's happening and start to rate limit or temporarily block these sources). Anyone know if there is an interface for automated downloading of the raw data? Is anyone involved in these projects on list? It looks like you can request the data manually at the moment. It'd also be good to discuss merging data from these projects into an upstream 'open-generalbadstuff-project'. Cheers, Robin
Re: [uknof] DNS/NTP censured, a solution !
As I have been asked off-line quite a few times : We wrote it to complement NFSEN. You can only search NFSEN once the data has been fully analysed. It mean that most of the time you have to wait a few minutes. We were seeing 15 mns DDOS, at least twice a day. By the time we had identified the DDOS pattern, it was off. This is what prompted the creation of ExaDDOS. Just to be able to see what was happening in that time and react faster. Thomas On 12 Feb 2014, at 16:57, Thomas Mangin thomas.man...@exa-networks.co.uk wrote: Hello, Because : - Exa has been under attack way too much these last weeks - We hate to have to deal with it Because: - Andrisoft seems cool but does not do FlowSpec - Arbor is known for its price (and features) - I am from Yorkshire (How much do you pay me to find bugs in your shinny application ?) Because: - We can ... - And people can not be bothered to fix the problem at source ! I have been working on making our internal tool ( Thank you Daniel ) something which can be built on and released to the community. The repository is here: https://github.com/Exa-Networks/exaddos The code is not even one week old but it can : - use SNMP to monitor your EBGP interfaces - parse IPFIX to find your top speakers - provide you the data in an HORRIBLE web page ( but all the rendering is client side, so feel free to fix that !) Now I would love some help ... I am NOT a web designer who find Javascript easy (I can handle jquery and basic stuff but nice CSS is not my cup of tea), so it will not look nice unless someone else make it so. I can provide the underlying data via JSON in whatever way one may need to allow : - graphing of links - allow to drill down on top speakers to find proto / ports information - one click get rid of that DDOS for IP proto I did some of this stuff with ExaProxy so I am not totally useless but god knows it is not my strength ! So any help would be welcome, so I can go back on coding on BGP and not DDOS. Thomas PS: I created a G+ community ExaDDOS .. I will try to add a mailing list later on. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [uknof] DNS/NTP censured, a solution !
Hi Thomas, Interesting timing - we've also been seeing a big increase in the same over the last few weeks, mainly targeting schools from automated ( cheap!) online 'booter' services (presumably instigated by students who have had enough of their IT lessons). We've also been forced to script something similar to analyse flows each minute and advertise blackholes upstream in an automated fashion in order to react quicker. I found that the complexity (and the bit I imagine the paid mitigation services spend a lot of their RD on) is the 'analysis' part to reliably detect. I found it easy enough for some of the simple attacks hitting us though. Our scripted version is very specific to the way we're set up so it wouldn't really translate elsewhere, but I'll be interested to take a look through your git repo. Alas, I'm no front-end/gui coder either :) One thing I did think would be useful while I was doing this, was if there was an 'open' online IP address reputation database (similar to a spam reputation db) - I couldn't find one with a quick Google. Seems to me it wouldn't take much for different providers all analysing flows to come up with a fairly reliable list of sources for some of this amplification attack traffic (provided the source isn't spoofed, which normally amplified stuff wouldn't be). Having that list to use when determining whether a flow I'm analysing is a DDoS (to use as a weighting amongst other factors) would help a lot, and could maybe even be used to drop such traffic in the network based on source rather than blackholing destinations upstream, provided the network could take the hit (though getting into a bit of neutrality debate there I guess!) Regards, Robin. On 12/02/14 19:05, Thomas Mangin wrote: As I have been asked off-line quite a few times : We wrote it to complement NFSEN. You can only search NFSEN once the data has been fully analysed. It mean that most of the time you have to wait a few minutes. We were seeing 15 mns DDOS, at least twice a day. By the time we had identified the DDOS pattern, it was off. This is what prompted the creation of ExaDDOS. Just to be able to see what was happening in that time and react faster. Thomas On 12 Feb 2014, at 16:57, Thomas Mangin thomas.man...@exa-networks.co.uk wrote: Hello, Because : - Exa has been under attack way too much these last weeks - We hate to have to deal with it Because: - Andrisoft seems cool but does not do FlowSpec - Arbor is known for its price (and features) - I am from Yorkshire (How much do you pay me to find bugs in your shinny application ?) Because: - We can ... - And people can not be bothered to fix the problem at source ! I have been working on making our internal tool ( Thank you Daniel ) something which can be built on and released to the community. The repository is here: https://github.com/Exa-Networks/exaddos The code is not even one week old but it can : - use SNMP to monitor your EBGP interfaces - parse IPFIX to find your top speakers - provide you the data in an HORRIBLE web page ( but all the rendering is client side, so feel free to fix that !) Now I would love some help ... I am NOT a web designer who find Javascript easy (I can handle jquery and basic stuff but nice CSS is not my cup of tea), so it will not look nice unless someone else make it so. I can provide the underlying data via JSON in whatever way one may need to allow : - graphing of links - allow to drill down on top speakers to find proto / ports information - one click get rid of that DDOS for IP proto I did some of this stuff with ExaProxy so I am not totally useless but god knows it is not my strength ! So any help would be welcome, so I can go back on coding on BGP and not DDOS. Thomas PS: I created a G+ community ExaDDOS .. I will try to add a mailing list later on.
Re: [uknof] DNS/NTP censured, a solution !
Hi Robin, On 12 Feb 2014, at 23:07, Robin Williams robin.willi...@tnp.net.uk wrote: Interesting timing - we've also been seeing a big increase in the same over the last few weeks, mainly targeting schools from automated ( cheap!) online 'booter' services (presumably instigated by students who have had enough of their IT lessons). Same here, our DDOS were as well very short, 15ms hence why our focus was on reacting to abnormal flows quickly (enough to stop the course, not enough to piss of the ISP ?). We found a way to disable the relation between the school and the control machine, since then we had no more attack. As it seems that attack must be initiated from inside the school (at least from what we have seen/understood). We've also been forced to script something similar to analyse flows each minute and advertise blackholes upstream in an automated fashion in order to react quicker. That's why open source is great, I hate when everyone is re-inventing the same wheel :-) I found that the complexity (and the bit I imagine the paid mitigation services spend a lot of their RD on) is the 'analysis' part to reliably detect. The trick we found is to detect when our upstream pass abnormal threshold and look at that time for the top speaker in terms of pps. Not perfect but as it still require someone here to pull the trigger it works pretty well. I found it easy enough for some of the simple attacks hitting us though. Our scripted version is very specific to the way we're set up so it wouldn't really translate elsewhere, but I'll be interested to take a look through your git repo. Alas, I'm no front-end/gui coder either :) Our production version written by Daniel is ahead of ExaBGP but then it is as well very specific. I intend to catch with him and have the noc team switch tool, but for the last days, I am told he added a feature I am still missing :p Perhaps I should force him to work on my code base :p I would be interested in sharing idea, but I guess it would be better to take the discussion off-list. One thing I did think would be useful while I was doing this, was if there was an 'open' online IP address reputation database (similar to a spam reputation db) - I couldn't find one with a quick Google. No - I do not know of any neither, but IMHO we are always fighting the problem the wrong way : ISP knows their customers and should be able to detect outgoing DDOS, instead everyone is paying big money to stop INCOMING flows. The other day we detected that we were part of the problem and stopped the traffic, funnily we got a mail from the recipient of the attack who was very surprised when we told him that (a) we knew (b) it had been sorted the day before. I tried to push the same idea with spam a few years back (and even wrote some proof of concept code with ScavengerEXA) but got nowhere ... However I may be able to bring back some of the idea in ExaDDOS if the project get traction. Seems to me it wouldn't take much for different providers all analysing flows to come up with a fairly reliable list of sources for some of this amplification attack traffic (provided the source isn't spoofed, which normally amplified stuff wouldn't be). hum .. a list of open NTP servers ? Having that list to use when determining whether a flow I'm analysing is a DDoS (to use as a weighting amongst other factors) would help a lot, and could maybe even be used to drop such traffic in the network based on source rather than blackholing destinations upstream, provided the network could take the hit (though getting into a bit of neutrality debate there I guess!) We do not blackhole the destination unless we have really no other choice as otherwise the attacker wins, hence why I like FlowSpec so much. Thomas signature.asc Description: Message signed with OpenPGP using GPGMail
