Re: BCP38.info
On Feb 3, 2014, at 2:55 PM, Dobbins, Roland rdobb...@arbor.net wrote: It would be useful to know whether there are in fact NATs, or are 'DNS forwarders' . . . Another question is whether or not it's possible that in at least some cases, MITMing boxes on intermediary networks are grabbing these queries and then spoofing the scanner source IP as they redirect the queries . . . . if this is taking place, then it would be the network(s) with the MITMing box(es) which allow spoofing, irrespective of whether or not the intended destination networks do, yes? --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: BCP38.info, RELATING: TWC (AS11351) blocking all NTP?
Hi, I think I might have already deleted subject matter a few days ago in re: BCP38. What exactly are you trying to do? I agree my general comment about the recent NTP weaknesses should be addressed via IPv6 RFC may have been mis-understood. I meant mostly that with IPv6 NAT goes away, all devices are exposed, and we also have the 'internet of things' - much more subject to potential abuse. An NTPv5 solution that could be done with NTP services already, and would be more of a 'best practices of how this shit starts up and what it can do' and educating vendors to have reasonable behavior in the first place? And an NTPv6 solution/RFC/guideline that was similar, could help? Neither will 'solve the problem' - but I think the idea of managing what somebody can do and having the provider filter in/out on IPv4 and/or mobile ipV4, much less ipV6 is very unorthodox and much against the spirit of having global m:n communications be helpful for humanity. My apologies if I mis-understand your recent and last few e-mails. I disagree that 'filtering' or 'blocking' any kind of IPv4 or IPv6 protocol to 'protect the end user' is the wrong way to go when compared to just having things work in a secure manner. - Mike On Feb 3, 2014, at 12:07 AM, Dobbins, Roland rdobb...@arbor.net wrote: On Feb 3, 2014, at 2:55 PM, Dobbins, Roland rdobb...@arbor.net wrote: It would be useful to know whether there are in fact NATs, or are 'DNS forwarders' . . . Another question is whether or not it's possible that in at least some cases, MITMing boxes on intermediary networks are grabbing these queries and then spoofing the scanner source IP as they redirect the queries . . . . if this is taking place, then it would be the network(s) with the MITMing box(es) which allow spoofing, irrespective of whether or not the intended destination networks do, yes? --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: BCP38.info, RELATING: TWC (AS11351) blocking all NTP?
On Feb 3, 2014, at 3:24 PM, Michael DeMan na...@deman.com wrote: I meant mostly that with IPv6 NAT goes away, I don't know if this is true or not - and even if it is true, it's going to be a long, long time before the IPv4 Internet goes away (like, maybe, pretty much forever, heh). An NTPv5 solution that could be done with NTP services already, and would be more of a 'best practices of how this shit starts up and what it can do' and educating vendors to have reasonable behavior in the first place? Yes, but that's many years away, and doesn't address legacy issues. And an NTPv6 solution/RFC/guideline that was similar, could help? Again, many years away, and doesn't address legacy issues. I disagree that 'filtering' or 'blocking' any kind of IPv4 or IPv6 protocol to 'protect the end user' is the wrong way to go when compared to just having things work in a secure manner. Yes, but since the latter part of this statement is unattainable in the foreseeable future, the idea is actually to protect *the rest of the Internet* from misconfigured CPE. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: TWC (AS11351) blocking all NTP?
On Sun, Feb 02, 2014 at 02:49:49PM -0800, Matthew Petach mpet...@netflight.com wrote a message of 49 lines which said: If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place. It is a bit more complicated. Reflection with amplification is certainly much less useful for an attacker but it has still some advantages: the attack traffic coming to the victim's AS will be distributed differently (entering via different peers), making tracking the attacker through Netflow/Ipfix more difficult.
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 03, 2014 at 04:09:39AM +, Dobbins, Roland rdobb...@arbor.net wrote a message of 20 lines which said: I also think that restricting your users by default to your own recursive DNS servers, plus a couple of well-known, well-run public recursive services, is a good idea - as long as you allow your users to opt out. That's a big as long. I agree with you but I'm fairly certain that most ISP who deny their users the ability to do DNS requests directly (or to run their own DNS resolver) have no such opt-out (or they make it expensive and/or complicated). After all, when outside DNS is blocked, it is more often for business reasons (forcing the users to use a local lying resolver, with ads when NXDOMAIN is returned) than for security reasons.
Re: TWC (AS11351) blocking all NTP?
On Feb 3, 2014, at 4:55 PM, Stephane Bortzmeyer bortzme...@nic.fr wrote: I agree with you but I'm fairly certain that most ISP who deny their users the ability to do DNS requests directly (or to run their own DNS resolver) have no such opt-out (or they make it expensive and/or complicated). There are some who do it, though, with a user-friendly portal - I've seen it in action. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: BCP38.info, RELATING: TWC (AS11351) blocking all NTP?
On Mon, 03 Feb 2014 00:24:08 -0800, Michael DeMan said: An NTPv5 solution that could be done with NTP services already Doesn't matter - the same people that aren't upgrading to a correctly configured NTPv4 aren't going to upgrade to an NTPv5. No need at all for a protocol increment (and actually doing that has its own issues). pgpReYols1Oli.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
How about this - I have proposed to NIST we start filtering - realize that the NIST ITS program itself was setup to run NTP in an open access mode - we host a dozen or so of those systems and so we get hit all the time. The solution is actually not running timing services across UDP because of the hand shaking over the open Internet - and that obviously isnt going to happen. My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. If this interests you contact me off list. Todd Glassey - USTiming.ORG On 2/2/2014 7:17 PM, ryang...@gmail.com wrote: I'd hate to think that NetOps would be so heavy handed in blocking all of UDP, as this would essentially halt quite a bit of audio/video traffic. That being said, there's still quite the need for protocol improvement when making use of UDP, but blocking UDP as a whole is definitely not a resolution, and simply creating a wall that not only keeps the abusive traffic out, but keeps legitimate traffic from flowing freely as it should. Sent on the TELUS Mobility network with BlackBerry -Original Message- From: Cb B cb.li...@gmail.com Date: Sun, 2 Feb 2014 15:09:55 To: Matthew Petachmpet...@netflight.com Cc: nanog@nanog.org Subject: Re: TWC (AS11351) blocking all NTP? On Feb 2, 2014 2:54 PM, Matthew Petach mpet...@netflight.com wrote: On Sun, Feb 2, 2014 at 2:17 PM, Cb B cb.li...@gmail.com wrote: On Feb 2, 2014 8:35 AM, Jonathan Towne jto...@slic.com wrote: The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region. And not just your provider, everyone is dealing with UDP amp attacks. These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not knee jerk, it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild. People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively. Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses. I dont want to go into fault, there is plenty of that to go around. If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place. I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue. Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on. My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/ My crystal ball says all of UDP will show up soon. CB Thanks! Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;) -- - Personal Email - Disclaimers Apply
Re: TWC (AS11351) blocking all NTP?
On Mon, 03 Feb 2014 06:14:30 -0800, TGLASSEY said: My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. That's going to be one big VLAN. /me makes popcorn. pgp0cVq4AACgv.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
Or a whole bunch of small ones Vladis - and yes we are capable of handling the loads. :-) Todd On 2/3/2014 6:34 AM, valdis.kletni...@vt.edu wrote: On Mon, 03 Feb 2014 06:14:30 -0800, TGLASSEY said: My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. That's going to be one big VLAN. /me makes popcorn. -- - Personal Email - Disclaimers Apply
Re: TWC (AS11351) blocking all NTP?
On Mon, 03 Feb 2014 06:50:56 -0800, TGLASSEY said: Or a whole bunch of small ones Vladis - and yes we are capable of handling the loads. 38,917 vlans later... /me makes even *more* popcorn... pgphM_JWCrh3v.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
On 2/3/14, 1:02 AM, Dobbins, Roland rdobb...@arbor.net wrote: All that broadband access operators need to do is to a) enforce antispoofing as close to their customers as possible, Many probably do so already. But as well all know, it only takes a few that don¹t to create a large scale issue. IMHO if we want to change this we need to (1) measure where and how anti-spoofing is not implemented, (2) contact networks that have not implemented with recommendations on how to do so (a la bcp38.info), and (3) failing reasonable response to #2 to start a public running list of networks that have inadequate levels of anti-spoofing (according to some reasonable standard, and probably more nuanced than a binary result). Jason
Do network diagnostic tools need upgrade?
Hello NANOG list members, I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, don't you think it's time to have an upgraded version of icmp protocol? from my side there is a lot that I can NOT do with current tools and protocols, here are few scenarios, and I would like to hear yours: First scenario: To be able to troubleshoot advanced networks with complex QoS and policy-based routing configuration, where ping, traceroute and other network diagnostic tools do not provide accurate readings, for example, you are troubleshooting a web server with ping, and it looks functioning quite well (packet loss and round trip time is all good), but web services are still significantly slow, the fact is icmp and tcp:80 might have different priorities and bandwidth limits on each router along the path between the client and the server, in this case, network admins usually use telnet applications like (Paping), well it may help if the forward and return path of all packets are exactly the same. Second scenario: So another possible scenario is that you need to determine readings for forward and return paths, TraceRoute for example gives you forward path only using icmp. But what if you need to troubleshoot a VoIP server for example, assuming that packets return path might not be the same as forward path. Third scenario: One of the most common problems in networking, is that you don't have access to all equipment between client and server, but you have to troubleshoot the path between them and to understand where the problem exactly is in order to contact the right person without having the privilege to check the configuration on each router. Fourth scenario: Also, with trace route you can't determine the actual path, for example, the router may direct http traffic to proxy server while leaving other traffic going through a different hop.
RE: Updated ARIN allocation information
Tore Anderson wrote: [...] It's not exactly new. Like I've mentioned earlier in this thread, the RIPE NCC has granted assignments smaller than /24 to requestors since, well, forever. There are currently 238 such assignments listed in delegated-ripencc-extended-latest.txt. However, these microscopic assignments have proven hugely unpopular, amounting for only a fraction of a percent of the total (there are 27733 assignments equal to or larger than /24 in the same file). If I remember correctly, while some (most?) people were disappointed by the lack of a minimum size for PI assignments, others valued the ability to register addresses for interfaces that needed to be unique but did not require global connectivity. Regards, Leo smime.p7s Description: S/MIME cryptographic signature
Re: BGP multihoming
* Tore Anderson * Baldur Norddahl Is assigning a /24 from my own PA space for the purpose of BGP multihoming considered sufficient need? Not with current policies, no That was then. With current policies: yes. To elaborate a bit, the RIPE Community just reached consensus on a policy change that makes the size and the purpose of an assignment entirely a local decision. That means that if you and your customer agree that a /X is needed for purpose Y, and you as the LIR have the available space and the willingness to make that assignment, you are now free to make it. The new IPV4 policy does not mandate any limits to what X and Y might be, except for the fact that Y must somehow involve «operating a network» (your use case certainly qualifies). Tore
BGP peer traffic monitoring
I have a router with about 20 peers, most are all on a single port (local exchange), how is everyone monitoring traffic to individual peers? Dennis Burgess, Mikrotik Certified Trainer Author of Learn RouterOS- Second Edition http://www.wlan1.com/product_p/mikrotik%20book-2.htm Link Technologies, Inc -- Mikrotik WISP Support Services Office: 314-735-0270 tel:314-735-0270 Website: http://www.linktechs.net http://www.linktechs.net/ - Skype: linktechs skype:linktechs?call -- Create Wireless Coverage's with www.towercoverage.com http://www.towercoverage.com/ - 900Mhz - LTE - 3G - 3.65 - TV Whitespace
Re: TWC (AS11351) blocking all NTP?
Why burn the village when only one house is the problem? I thought
there might be some interest in hearing about work being done to use
SDN to automatically configure filtering in existing switches and
routers to mitigate flood attacks.
Real-time analytics based on measurements from switches/routers
(sFlow/PSAMP/IPFIX) can identify large UDP flows and integrated hybrid
OpenFlow, I2RS, REST, NETCONF APIs, etc. can be used to program the
switches/routers to selectively filter traffic based on UDP port and
IP source / destination. By deploying a DDoS mitigation SDN
application, providers can use their existing infrastructure to
protect their own and their customers networks from flood attacks, and
generate additional revenue by delivering flood protection as a value
added service.
https://datatracker.ietf.org/doc/draft-krishnan-i2rs-large-flow-use-case/
http://events.linuxfoundation.org/sites/events/files/slides/flow-aware-real-time-sdn-analytics-odl-summit-v2.pdf
Specifically looking at sFlow, large flood attacks can be detected
within a second. The following article describes a simple example
using integrated hybrid OpenFlow in a 10/40G ToR switch:
http://blog.sflow.com/2014/01/physical-switch-hybrid-openflow-example.html
The example can be modified to target NTP mon_getlist requests and
responses using the following sFlow-RT flow definition:
{'ipdestination,udpsourceport',value:'ntppvtbytes',filter:'ntppvtreq=20,42'}
or to target DNS ANY requests:
{keys:'ipdestination,udpsourceport',value:'frames',filter:'dnsqr=truednsqtype=255'}
The OpenFlow block control can be modified to selectively filter UDP
traffic based on the identified UDP source port and destination IP
address.
Vendors are adding new SDN capabilities to their platforms (often as
software upgraded), so it's worth taking a look and seeing what is
possible.
Peter
On Sun, Feb 2, 2014 at 7:38 PM, Larry Sheldon larryshel...@cox.net wrote:
On 2/2/2014 9:17 PM, ryang...@gmail.com wrote:
I'd hate to think that NetOps would be so heavy handed in blocking
all of UDP, as this would essentially halt quite a bit of audio/video
traffic. That being said, there's still quite the need for protocol
improvement when making use of UDP, but blocking UDP as a whole is
definitely not a resolution, and simply creating a wall that not only
keeps the abusive traffic out, but keeps legitimate traffic from
flowing freely as it should.
We had to burn down the village to save it.
--
Requiescas in pace o email Two identifying characteristics
of System Administrators:
Ex turpi causa non oritur actio Infallibility, and the ability to
learn from their mistakes.
(Adapted from Stephen Pinker)
Re: BGP peer traffic monitoring
On Mon, Feb 03, 2014 at 11:48:04AM -0600, Dennis Burgess wrote: I have a router with about 20 peers, most are all on a single port (local exchange), how is everyone monitoring traffic to individual peers? Use something like IPFIX, NetFlow, sFlow and take a look at these two tools: http://pmacct.net/ https://github.com/manuelkasper/AS-Stats -- Kind regards, Job pgpgMF7V9C8xT.pgp Description: PGP signature
Re: Do network diagnostic tools need upgrade?
Oldies, but goodies: shaperprobe (1st), pchar (3rd), tcptrace.org, lft (4th), iftop, nsping (2nd), iperf, sjitter, pathneck (3rd) These are newer -- http://www.internet2.edu/products-services/performance-monitoring/performance-tools/ (OWAMP, 2nd) -- http://paris-traceroute.net (4th) -- http://packetdrill.googlecode.com dre On Mon, Feb 3, 2014 at 4:33 PM, Ammar Salih ammar.alsa...@gmail.com wrote: Hello NANOG list members, I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, don't you think it's time to have an upgraded version of icmp protocol? from my side there is a lot that I can NOT do with current tools and protocols, here are few scenarios, and I would like to hear yours: First scenario: To be able to troubleshoot advanced networks with complex QoS and policy-based routing configuration, where ping, traceroute and other network diagnostic tools do not provide accurate readings, for example, you are troubleshooting a web server with ping, and it looks functioning quite well (packet loss and round trip time is all good), but web services are still significantly slow, the fact is icmp and tcp:80 might have different priorities and bandwidth limits on each router along the path between the client and the server, in this case, network admins usually use telnet applications like (Paping), well it may help if the forward and return path of all packets are exactly the same. Second scenario: So another possible scenario is that you need to determine readings for forward and return paths, TraceRoute for example gives you forward path only using icmp. But what if you need to troubleshoot a VoIP server for example, assuming that packets return path might not be the same as forward path. Third scenario: One of the most common problems in networking, is that you don't have access to all equipment between client and server, but you have to troubleshoot the path between them and to understand where the problem exactly is in order to contact the right person without having the privilege to check the configuration on each router. Fourth scenario: Also, with trace route you can't determine the actual path, for example, the router may direct http traffic to proxy server while leaving other traffic going through a different hop.
Re: Do network diagnostic tools need upgrade?
On Mon, 03 Feb 2014 16:33:34 +0300, Ammar Salih said: I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, don't you think it's time to have an upgraded version of icmp protocol? from my side there is a lot that I can NOT do with current tools and protocols, here are few scenarios, and I would like to hear yours: Upgrading ICMP protocol is... challenging. I wouldn't even bother with trying to do anything with IPv4. Might be some options for IPv6, *IF* you can provide a *specific* proposal that looks worth the added code and router complexity Also, remember that most routers will do packet forwarding in hardware if they can just suck in bits on one interface and toss them out another - but if they have to do stuff like create and send an ICMP TTL Exceeded packet, you end up on the control plane and probably rate-limited. applications like (Paping), well it may help if the forward and return path of all packets are exactly the same. That's a routing problem, not an ICMP problem. As are the remainder of your examples. pgpp_Kutwos0F.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 12:42 PM, Peter Phaal peter.ph...@gmail.com wrote:
Why burn the village when only one house is the problem? I thought
there might be some interest in hearing about work being done to use
SDN to automatically configure filtering in existing switches and
routers to mitigate flood attacks.
that's great... who's got sdn capable gear in deployments today? with
code and OSS stuff to deal with random SDN pokery? and who has spare
tcam/etc to deal with said pokery of 'block the attack-du-jour' ?
There's certainly the case that you could drop acls/something on
equipment to selectively block the traffic that matters... I suspect
in some cases the choice was: 50% of the edge box customers on this
location are a problem, block it across the board here instead of X00
times (see concern about tcam/etc problems)
Real-time analytics based on measurements from switches/routers
(sFlow/PSAMP/IPFIX) can identify large UDP flows and integrated hybrid
OpenFlow, I2RS, REST, NETCONF APIs, etc. can be used to program the
switches/routers to selectively filter traffic based on UDP port and
IP source / destination. By deploying a DDoS mitigation SDN
application, providers can use their existing infrastructure to
protect their own and their customers networks from flood attacks, and
generate additional revenue by delivering flood protection as a value
added service.
yup, that sounds wonderous... and I'm sure that in the future utopian
world (like 7-10 years from now, based on age-out of gear and OSS IT
change requirements) we'll see more of this. I don't think you'll see
much (in terms of edge ports on the network today) of this happening
'right now' though.
https://datatracker.ietf.org/doc/draft-krishnan-i2rs-large-flow-use-case/
http://events.linuxfoundation.org/sites/events/files/slides/flow-aware-real-time-sdn-analytics-odl-summit-v2.pdf
Specifically looking at sFlow, large flood attacks can be detected
within a second. The following article describes a simple example
using integrated hybrid OpenFlow in a 10/40G ToR switch:
hopefully there's some clamp on how much change per device/port you
plan too? :) I'd hate to see the RP/RE/etc get so busy programming
tcam that bgp/isis/ospf/etc flaps :(
http://blog.sflow.com/2014/01/physical-switch-hybrid-openflow-example.html
The example can be modified to target NTP mon_getlist requests and
responses using the following sFlow-RT flow definition:
{'ipdestination,udpsourceport',value:'ntppvtbytes',filter:'ntppvtreq=20,42'}
or to target DNS ANY requests:
{keys:'ipdestination,udpsourceport',value:'frames',filter:'dnsqr=truednsqtype=255'}
this also assume almost 1:1 sampling... which might not be feasible
either...otherwise you'll be seeing fairly lossy results, right?
The OpenFlow block control can be modified to selectively filter UDP
traffic based on the identified UDP source port and destination IP
address.
hopefully your OSS and netflow/sflow collection isn't also being used
for traffic engineering/capacity planning purposes? else... you might
get odd results from that infrastructure with such changes to the
sflow/netflow sender platform.
Vendors are adding new SDN capabilities to their platforms (often as
software upgraded), so it's worth taking a look and seeing what is
possible.
the device side is PROBABLY the simple side of the equation for most people...
On Sun, Feb 2, 2014 at 7:38 PM, Larry Sheldon larryshel...@cox.net wrote:
On 2/2/2014 9:17 PM, ryang...@gmail.com wrote:
I'd hate to think that NetOps would be so heavy handed in blocking
all of UDP, as this would essentially halt quite a bit of audio/video
traffic. That being said, there's still quite the need for protocol
improvement when making use of UDP, but blocking UDP as a whole is
definitely not a resolution, and simply creating a wall that not only
keeps the abusive traffic out, but keeps legitimate traffic from
flowing freely as it should.
We had to burn down the village to save it.
--
Requiescas in pace o email Two identifying characteristics
of System Administrators:
Ex turpi causa non oritur actio Infallibility, and the ability to
learn from their mistakes.
(Adapted from Stephen Pinker)
BCP38 [Was: Re: TWC (AS11351) blocking all NTP?]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/2/2014 2:17 PM, Cb B wrote: And, i agree bcp38 would help but that was published 14 years ago. But what? Are you somehow implying that because BCP38 was ...published 14 years ago (RFC2267 was initially published in 1998, and it was subsequently replaced by RFC2827)? I hope not, because BCP38 filtering would still help stop spoofed traffic now perpetuating these attacks, 14 years after BCP38 was published, because spoofing is at the root of this problem (reflection/amplification attacks). This horse is not dead, and still deserves a lot of kicking. $.02, - - ferg (co-author of BCP38) - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLv3ocACgkQKJasdVTchbLhowEAuO9DSQiRswVeqpHSccHo060h cqmIB8XlaNkzEPQw1w0A/0G6cjvtWBiJfwWbWoTY7X3RRMHeN36RkYR+2TonyNBi =W2wU -END PGP SIGNATURE-
Re: TWC (AS11351) blocking all NTP?
In regards to anti-spoofing measures - I think there a couple of vectors about the latest NTP attack where more rigorous client-side anti-spoofing could help but will not solve it overall. Most NTP servers only send legitimate traffic to a handful of masters, often in the ntp.org pool, and to peers and clients on their own network. I know that when I adjusted my NTP config to stop responding to traffic other than its ntp.org masters and the local LAN, the outbound DDoS traffic stopped. It took a while for the bad guys to notice, so I added some packet filters to limit the load on the NTP daemon. It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. R's, John
RE: BGP peer traffic monitoring
We perform MAC Based accounting on our IX interface and that allows us to monitor / graph traffic based off MAC address instead of being limited to the aggregate data of a single interface. Here's the JUNOS way of doing it, I'm sure other vendors have their equivalent. http://www.juniper.net/techpubs/en_US/junos13.2/topics/usage-guidelines/interfaces-configuring-mac-address-accounting.html JJ Stonebraker IP Network Engineering Grande Communications 512.878.5627 -Original Message- From: Dennis Burgess [mailto:dmburg...@linktechs.net] Sent: Monday, February 03, 2014 11:48 AM To: NANOG list Subject: BGP peer traffic monitoring I have a router with about 20 peers, most are all on a single port (local exchange), how is everyone monitoring traffic to individual peers? Dennis Burgess, Mikrotik Certified Trainer Author of Learn RouterOS- Second Edition http://www.wlan1.com/product_p/mikrotik%20book-2.htm Link Technologies, Inc -- Mikrotik WISP Support Services Office: 314-735-0270 tel:314-735-0270 Website: http://www.linktechs.net http://www.linktechs.net/ - Skype: linktechs skype:linktechs?call -- Create Wireless Coverage's with www.towercoverage.com http://www.towercoverage.com/ - 900Mhz - LTE - 3G - 3.65 - TV Whitespace
Re: TWC (AS11351) blocking all NTP?
On 03 Feb 2014 18:23:31 +, John Levine said: It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. You have that backwards - you can (possibly) safely filter it if you can establish *for certain* that it *isn't* in the ntp.org pools. pgpKhXjJIcqWo.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
On Feb 3, 2014, at 12:45 AM, Michael DeMan na...@deman.com wrote: The recently publicized mechanism to leverage NTP servers for amplified DoS attacks is seriously effective. I had a friend who had a local ISP affected by this Thursday and also another case where just two asterisk servers saturated a 100mbps link to the point of unusability. Once more - this exploit is seriously effective at using bandwidth by reflection. The challenge I see is there's some hosts like this one: [jared@nowherelikehome ]$ ntpq -c rv 111.107.252.142 associd=0 status=06f4 leap_none, sync_ntp, 15 events, freq_mode, version=ntpd 4.2.0-r Fri Jul 22 09:50:16 JST 2011 (1), processor=seil5, system=NetBSD/3.1_STABLE, leap=00, stratum=5, precision=-18, rootdelay=9.138, rootdispersion=132.247, peer=58012, refid=172.22.203.213, reftime=d685a094.9c806290 Sun, Jan 19 2014 0:53:40.611, poll=10, clock=d69a5d3c.c6b1a2a4 Mon, Feb 3 2014 18:23:56.776, state=4, offset=-0.598, frequency=-1.463, jitter=0.229, stability=0.042 This host will happily generate 100GB response to a single packet. They even have advisories posted: http://www.seil.jp/support/security/a01411.html Getting the information into the admin is hard. Time zones, language barriers, folks understanding why having unmaintained NTP hosts out there can be a significant issue. We found many ILO/IPMI interfaces that have NTP you can't do anything about (no filters, etc) - let alone patch .. Through ACL (hopefully not) or folks fixing hosts the following trend is observable in # of unique hosts that respond to NTP packets: 1529866 2014-01-10 1402569 2014-01-17 803156 2014-01-24 564027 2014-01-31 I will say that an awful lot of firewall operators out there seem to now be saying NTP BAD and generating panic'ed emails about NTP traffic. - Jared
Re: [NANOG-announce] NANOG On The Road - San Diego
This event sounds like a lot of fun, and I look forward to attending. :) Just curious if anyone wants to participate in an informal PGP key signing activity while we're there? I'm thinking an old fashioned everyone brings their own slips of paper type thing, but if there is sufficient interest I wouldn't mind helping to coordinate a key ring and printed sheets ... Doug On 01/29/2014 07:17 PM, Betty Burke be...@nanog.org wrote: Colleagues: In partnership, the American Registry for Internet Numbers (ARIN) and the North American Network Operators Group (NANOG) will bring ARIN+NANOG on the Road to San Diego, CA on Tuesday, February 25, 2014. The one day event will be held at the Handerly Hotel Resort. Please pass along this information to those who would benefit from attending a free event featuring educational sessions, great discussions, and networking opportunities! Morning sessions will get attendees up to speed on Internet number resource management, ARIN technical services, current policy discussions and more. Afternoon sessions will feature NANOG Tutorials on Network Security, IPv6, Traceroutes, BGB, including an update on the NANOG Best Current Operational Practice (BCOP) project. Complete information and meeting links are available on the NANOG websitehttps://www.nanog.org/meetings/road2/home . Should you have any questions, please feel free to send them to nanog-supp...@nanog.org. Sincerely, Betty
Re: Do network diagnostic tools need upgrade?
On 02/03/2014 05:33 AM, Ammar Salih wrote: Hello NANOG list members, I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, What tools are you referring to by ...? There are many others. I like tcptraceroute (there are two variants of it) and mtr. don't you think it's time to have an upgraded version of icmp protocol? What is it that you are thinking? ICMP is for signaling between machines. Increasing signaling for human diagnostics can lead to reconnaissance attacks. We don't want yet another option for some to incorrectly block ICMP [1], which in turn leads to other problems. [1] ... when they want to just block ICMP echo and reply, which is also bad enough and must be done really selectively. First scenario: To be able to troubleshoot advanced networks with complex QoS and policy-based routing configuration, where ping, traceroute and other network diagnostic tools do not provide accurate readings, for example, you are troubleshooting a web server with ping, and it looks functioning quite well (packet loss and round trip time is all good), but web services are still significantly slow, the fact is icmp and tcp:80 might have different priorities and bandwidth limits on each router along the path between the client and the server, in this case, network admins usually use telnet applications like (Paping), well it may help if the forward and return path of all packets are exactly the same. tcptraceroute. Second scenario: So another possible scenario is that you need to determine readings for forward and return paths, TraceRoute for example gives you forward path only using icmp. But what if you need to troubleshoot a VoIP server for example, assuming that packets return path might not be the same as forward path. It depends. Asymmetric routing is not necessarily bad unless it causes a problem like packet loss, high latency, etc. For example, if the return path has packet loss but you should 'hopefully' (yeah I know...) notice it in the traceroute if you increment the probe count or run it twice. Or try mtr, a periodic traceroute with different statistics presentation that significantly reduces the 'hopefully' problem. Third scenario: One of the most common problems in networking, is that you don't have access to all equipment between client and server, but you have to troubleshoot the path between them and to understand where the problem exactly is in order to contact the right person without having the privilege to check the configuration on each router. This one's more difficult but also it depends. State a specific problem case. Fourth scenario: Also, with trace route you can't determine the actual path, for example, the router may direct http traffic to proxy server while leaving other traffic going through a different hop. tcptraceroute.
Re: TWC (AS11351) blocking all NTP?
Huh? The issue with NTP relates to the monlist command (and a few others). These are management queries, and are not critical to the operation of a NTP server. You can disable these quite easily, and still run a NTP server that provides accurate time services. On 2/3/2014 9:14 AM, TGLASSEY wrote: How about this - I have proposed to NIST we start filtering - realize that the NIST ITS program itself was setup to run NTP in an open access mode - we host a dozen or so of those systems and so we get hit all the time. The solution is actually not running timing services across UDP because of the hand shaking over the open Internet - and that obviously isnt going to happen. My suggestion is that for those that need access we set up VLAN trunked private networking models to your ISP MPOE as it were in a digital context. If this interests you contact me off list. Todd Glassey - USTiming.ORG On 2/2/2014 7:17 PM, ryang...@gmail.com wrote: I'd hate to think that NetOps would be so heavy handed in blocking all of UDP, as this would essentially halt quite a bit of audio/video traffic. That being said, there's still quite the need for protocol improvement when making use of UDP, but blocking UDP as a whole is definitely not a resolution, and simply creating a wall that not only keeps the abusive traffic out, but keeps legitimate traffic from flowing freely as it should. Sent on the TELUS Mobility network with BlackBerry -Original Message- From: Cb B cb.li...@gmail.com Date: Sun, 2 Feb 2014 15:09:55 To: Matthew Petachmpet...@netflight.com Cc: nanog@nanog.org Subject: Re: TWC (AS11351) blocking all NTP? On Feb 2, 2014 2:54 PM, Matthew Petach mpet...@netflight.com wrote: On Sun, Feb 2, 2014 at 2:17 PM, Cb B cb.li...@gmail.com wrote: On Feb 2, 2014 8:35 AM, Jonathan Towne jto...@slic.com wrote: The provider has kindly acknowledged that there is an issue, and are working on a resolution. Heads up, it may be more than just my region. And not just your provider, everyone is dealing with UDP amp attacks. These UDP based amp attacks are off the charts. Wholesale blocking of traffic at the protocol level to mitigate 10s to 100s of gigs of ddos traffic is not knee jerk, it is the right thing to do in a world where bcp 38 is far from universal and open dns servers, ntp, chargen, and whatever udp 172 is run wild. People who run networks know what it takes to restore service. And increasingly, that will be clamping ipv4 UDP in the plumbing, both reactively and proactively. Please note that it's not that UDP is at fault here; it's applications that are structured to respond to small input packets with large responses. I dont want to go into fault, there is plenty of that to go around. If NTP responded to a single query with a single equivalently sized response, its effectiveness as a DDoS attack would be zero; with zero amplification, the volume of attack traffic would be exactly equivalent to the volume of spoofed traffic the originator could send out in the first place. I agree the source obfuscation aspect of UDP can be annoying, from the spoofing perspective, but that really needs to be recognized to be separate from the volume amplification aspect, which is an application level issue, not a protocol level issue. Source obfuscation is not annoying. Combined with amplification, it is the perfect storm for shutting down networks... whereby the only solution is to shutdown ipv4 udp. Or wave the magic wand that makes bcp38 universal, patches boxes, and so on. My point is: dont expect these abbused services on UDP to last. We have experience in access networks on how to deal with abused protocols. Here is one reference http://customer.comcast.com/help-and-support/internet/list-of-blocked-ports/ My crystal ball says all of UDP will show up soon. CB Thanks! Matt PS--yes, I know it would completely change the dynamics of the internet as we know it today to shift to a 1:1 correspondence between input requests and output replies...but it *would* have a nice side effect of balancing out traffic ratios in many places, altering the settlement landscape in the process. ;)
Re: Do network diagnostic tools need upgrade?
On Feb 3, 2014, at 1:59 PM, Octavio Alvarez alvar...@alvarezp.ods.org wrote: On 02/03/2014 05:33 AM, Ammar Salih wrote: Hello NANOG list members, I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, What tools are you referring to by ...? There are many others. I like tcptraceroute (there are two variants of it) and mtr. There are lesser known options that are used by folks, eg: ping record-route. One could certainly use those available tools, but most folks have a hard enough time interpreting traceroute output. I've seen customers complain about performance to have us show them it's on their network, or their firewall modules, etc.. Having statistics on network usage/errors/drops is incredible useful in isolating the performance limitations. Knowing that a firewall maxes at 350Mb/s is as equally useful as having protocol extensions to collect the data. One of my early experiences with a sysadmin who only cared about the application/OS was the router is a black box that gets my packets there. Knowing the behavior beyond there is also important (how latency/loss impacts tcp/udp/application performance for example). Most importantly, keeping an open mind when troubleshooting is helpful. Sometimes you find something unexpected. (eg: uRPF drops when responding IP is mapped-v4-in-v6 from within 6PE network). - Jared
Re: TWC (AS11351) blocking all NTP?
On Feb 4, 2014, at 12:42 AM, Peter Phaal peter.ph...@gmail.com wrote: Real-time analytics based on measurements from switches/routers (sFlow/PSAMP/IPFIX) can identify large UDP flows and integrated hybrid OpenFlow, I2RS, REST, NETCONF APIs, etc. can be used to program the switches/routers to selectively filter traffic based on UDP port and IP source / destination. By deploying a DDoS mitigation SDN application, providers can use their existing infrastructure to protect their own and their customers networks from flood attacks, and generate additional revenue by delivering flood protection as a value added service. This is certainly a general capability set towards which many operators are evolving (and it's always amusing how you leave out NetFlow, which many operators use, but include sFlow, which very few operators use, heh), but it's going to be quite some time before this sort of thing is practical and widely-deployale. Believe me, I've been working towards this vision for many years. It isn't going to happen overnight. Specifically looking at sFlow, large flood attacks can be detected within a second. And with NetFlow, and with IPFIX - the first of which is widely deployed today, and the second of which will be widely deployed in future. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: TWC (AS11351) blocking all NTP?
It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. Speaking as one of the 3841 servers in the pool.ntp.org pool, I'm happy to be described as a handful, something my mother used to say, but I do feel obligated to point out that it's a pretty big handful especially if you want to be fiddling ACLs on an hourly basis which is pretty much what it takes. And, of course, if you're one of that handful, then you've pretty much got to allow that NTP traffic in, although you're also probably, hopefully, clue-full enough not to let random hosts make you a DDoS accelerator. (the other) jms -- Joel M Snyder, 1404 East Lind Road, Tucson, AZ, 85719 Senior Partner, Opus One Phone: +1 520 324 0494 j...@opus1.comhttp://www.opus1.com/jms
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 10:16 AM, Christopher Morrow
morrowc.li...@gmail.com wrote:
On Mon, Feb 3, 2014 at 12:42 PM, Peter Phaal peter.ph...@gmail.com wrote:
Why burn the village when only one house is the problem? I thought
there might be some interest in hearing about work being done to use
SDN to automatically configure filtering in existing switches and
routers to mitigate flood attacks.
that's great... who's got sdn capable gear in deployments today? with
code and OSS stuff to deal with random SDN pokery? and who has spare
tcam/etc to deal with said pokery of 'block the attack-du-jour' ?
There's certainly the case that you could drop acls/something on
equipment to selectively block the traffic that matters... I suspect
in some cases the choice was: 50% of the edge box customers on this
location are a problem, block it across the board here instead of X00
times (see concern about tcam/etc problems)
I agree that managing limited TCAM space is critical to the
scaleability of any mitigation solution. However, tying up TCAM space
on every edge device with filters to prevent each new threat is likely
to be less scaleable than a measurement driven control that only takes
a TCAM slot on a device when an active attack is detected transiting
that device.
Real-time analytics based on measurements from switches/routers
(sFlow/PSAMP/IPFIX) can identify large UDP flows and integrated hybrid
OpenFlow, I2RS, REST, NETCONF APIs, etc. can be used to program the
switches/routers to selectively filter traffic based on UDP port and
IP source / destination. By deploying a DDoS mitigation SDN
application, providers can use their existing infrastructure to
protect their own and their customers networks from flood attacks, and
generate additional revenue by delivering flood protection as a value
added service.
yup, that sounds wonderous... and I'm sure that in the future utopian
world (like 7-10 years from now, based on age-out of gear and OSS IT
change requirements) we'll see more of this. I don't think you'll see
much (in terms of edge ports on the network today) of this happening
'right now' though.
The current 10G upgrade cycle provides an opportunity to deploy
equipment that is SDN capable. The functionality required for this use
case is supported by current generation merchant silicon and is widely
available right now in inexpensive switches.
Specifically looking at sFlow, large flood attacks can be detected
within a second. The following article describes a simple example
using integrated hybrid OpenFlow in a 10/40G ToR switch:
hopefully there's some clamp on how much change per device/port you
plan too? :) I'd hate to see the RP/RE/etc get so busy programming
tcam that bgp/isis/ospf/etc flaps :(
With integrated hybrid OpenFlow, there is very little activity on the
OpenFlow control plane. The normal BGP, ECMP, LAG, etc. control planes
handles forwarding of packets. OpenFlow is only used to selectively
override specific FIB entries.
I2RS provides a similar capability to selectively override RIB entries
and implement controls. However, I don't know if any vendors are
shipping I2RS capable routers today.
Typical networks probably only see a few DDoS attacks an hour at the
most, so pushing a few rules an hour to mitigate them should have
little impact on the switch control plane.
A good working definition of a large flow is 10% of a link's
bandwidth. If you only trigger actions for large flows then in the
worst case you would only require 10 rules per port to change how
these flows are treated.
http://blog.sflow.com/2014/01/physical-switch-hybrid-openflow-example.html
The example can be modified to target NTP mon_getlist requests and
responses using the following sFlow-RT flow definition:
{'ipdestination,udpsourceport',value:'ntppvtbytes',filter:'ntppvtreq=20,42'}
or to target DNS ANY requests:
{keys:'ipdestination,udpsourceport',value:'frames',filter:'dnsqr=truednsqtype=255'}
this also assume almost 1:1 sampling... which might not be feasible
either...otherwise you'll be seeing fairly lossy results, right?
Actually, to detect large flows (defined as 10% of link bandwidth)
within a second, you would only require the following sampling rates:
1G link, sampling rate = 1-in-1,000 (large flow = 100M bit/s)
10G link, sampling rate = 1-in-10,000 (large flow = 1G bit/s)
40G link, sampling rate = 1-in-40,000 (large flow = 4G bit/s
100G link, sampling rate = 1-in-100,000 (large flow = 10G bit/s)
These sampling rates are realistically achievable in production
networks (enabling monitoring on all ports) and would allow you to
detect the specific IP destination and UDP source port associated with
a flood attack, and the switches in the attack path, within a second.
The OpenFlow block control can be modified to selectively filter UDP
traffic based on the identified UDP source port and destination IP
address.
hopefully your OSS and netflow/sflow collection isn't
Re: TWC (AS11351) blocking all NTP?
On Feb 4, 2014, at 12:11 AM, Brian Rak b...@gameservers.com wrote: You can disable these quite easily, and still run a NTP server that provides accurate time services. Concur 100% - although it should be noted that 1:1 reflection without any amplification is also quite useful to attackers. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: TWC (AS11351) blocking all NTP?
On 2/3/2014 2:46 PM, Dobbins, Roland wrote: On Feb 4, 2014, at 12:11 AM, Brian Rak b...@gameservers.com wrote: You can disable these quite easily, and still run a NTP server that provides accurate time services. Concur 100% - although it should be noted that 1:1 reflection without any amplification is also quite useful to attackers. That's true, but there are countless services out there that could be abused in such a way. It's pretty much the same issue with DNS, even authoritative-only servers can be abused for reflection. Securing everything that could possibly be used for reflection is going to be a long and painful process, preventing this specific amplification attack is pretty easy. NTP clients have a long history of poor implementations, so the server already has rate limiting built in. While rate limiting outgoing replies isn't a perfect solution, it's significantly better then no rate limiting (for the curious, add 'limited' to your 'restrict default' lines to enable rate limiting. This doesn't help with the current amplification issues, but will help should someone just be abusing NTP servers for reflection).
Re: TWC (AS11351) blocking all NTP?
On Feb 4, 2014, at 3:09 AM, Brian Rak b...@gameservers.com wrote: It's pretty much the same issue with DNS, even authoritative-only servers can be abused for reflection. They are, every minute of every day - and they provide amplification, too. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: Do network diagnostic tools need upgrade?
I like observium for monitoring gear, tons of information, great way to find erroring fiber over thousands of devices and caught some memory leaks prior to impacting things.This is in addition to flow data of course. Bryan DigitalOcean We're Hiring
Re: TWC (AS11351) blocking all NTP?
It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. Speaking as one of the 3841 servers in the pool.ntp.org pool, I'm happy to be described as a handful, something my mother used to say, but I do feel obligated to point out that it's a pretty big handful especially if you want to be fiddling ACLs on an hourly basis which is pretty much what it takes. I was thinking that the ntp.org servers on any particular network are a small set of exceptions to a general rule to rate limit outgoing NTP traffic. Regards, John Levine, jo...@iecc.com, Primary Perpetrator of The Internet for Dummies, Please consider the environment before reading this e-mail. http://jl.ly
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 2:42 PM, Peter Phaal peter.ph...@gmail.com wrote: On Mon, Feb 3, 2014 at 10:16 AM, Christopher Morrow morrowc.li...@gmail.com wrote: On Mon, Feb 3, 2014 at 12:42 PM, Peter Phaal peter.ph...@gmail.com wrote: There's certainly the case that you could drop acls/something on equipment to selectively block the traffic that matters... I suspect in some cases the choice was: 50% of the edge box customers on this location are a problem, block it across the board here instead of X00 times (see concern about tcam/etc problems) I agree that managing limited TCAM space is critical to the scaleability of any mitigation solution. However, tying up TCAM space on every edge device with filters to prevent each new threat is likely yup, there's a tradeoff, today it's being made one way, tomorrow perhaps a different way. My point was that today the percentage of sdn capable devices is small enough that you still need a decimal point to measure it. (I bet, based on total devices deployed) The percentage of oss backend work done to do what you want is likely smaller... the folk in NZ-land (Citylink, reannz ... others - find josh baily / cardigan) are making some strides, but only in the exchange areas so far. fun stuff... but not the deployed gear as an L2/L3 device in TWC/Comcast/Verizon. Real-time analytics based on measurements from switches/routers (sFlow/PSAMP/IPFIX) can identify large UDP flows and integrated hybrid OpenFlow, I2RS, REST, NETCONF APIs, etc. can be used to program the switches/routers to selectively filter traffic based on UDP port and IP source / destination. By deploying a DDoS mitigation SDN application, providers can use their existing infrastructure to protect their own and their customers networks from flood attacks, and generate additional revenue by delivering flood protection as a value added service. yup, that sounds wonderous... and I'm sure that in the future utopian world (like 7-10 years from now, based on age-out of gear and OSS IT change requirements) we'll see more of this. I don't think you'll see much (in terms of edge ports on the network today) of this happening 'right now' though. The current 10G upgrade cycle provides an opportunity to deploy by 'current 10g upgrade cycle' you mean the one that happened 2-5 yrs ago? or somethign newer? did you mean 100G? equipment that is SDN capable. The functionality required for this use case is supported by current generation merchant silicon and is widely available right now in inexpensive switches. right... and everyone is removing their vendor supported gear and replacing it with pica8 boxes? The reality is that as speeds/feeds have increased over the last while basic operations techiques really haven't. Should they? maybe? will they? probably? is that going to happen on a dime? nope. Again, I suspect you'll see smaller deployments of sdn-like stuff 'soon' and larger deployments when people are more comfortable with the operations/failure modes that change. Specifically looking at sFlow, large flood attacks can be detected within a second. The following article describes a simple example using integrated hybrid OpenFlow in a 10/40G ToR switch: hopefully there's some clamp on how much change per device/port you plan too? :) I'd hate to see the RP/RE/etc get so busy programming tcam that bgp/isis/ospf/etc flaps :( With integrated hybrid OpenFlow, there is very little activity on the OpenFlow control plane. The normal BGP, ECMP, LAG, etc. control planes handles forwarding of packets. OpenFlow is only used to selectively override specific FIB entries. that didn't really answer the question :) if I have 10k customers behind the edge box and some of them NOW start being abused, then more later and that mix changes... if it changes a bunch because the attacker is really attackers. how fast do I change before I can't do normal ops anymore? Typical networks probably only see a few DDoS attacks an hour at the most, so pushing a few rules an hour to mitigate them should have little impact on the switch control plane. based on what math did you get 'few per hour?' As an endpoint (focal point) or as a contributor? The problem that started this discussion was being a contributor...which I bet happens a lot more often than /few an hour/. A good working definition of a large flow is 10% of a link's bandwidth. If you only trigger actions for large flows then in the worst case you would only require 10 rules per port to change how these flows are treated. 10% of a 1g link is 100mbps, For contributors to ntp attacks, many of the contributors are sending ONLY 300x the input, so less than 100mbps. On a 10g link it's 1G... even more hidden. This math and detection aren't HARD, but tuning it can be a bit challenging. http://blog.sflow.com/2014/01/physical-switch-hybrid-openflow-example.html The example can be modified to target NTP mon_getlist requests and responses using
Re: TWC (AS11351) blocking all NTP?
On Feb 3, 2014, at 3:29 PM, John R. Levine jo...@iecc.com wrote: It seems thata hosts sending large amounts of NTP traffic over the public Internet can be safely filtered if you don't already know that it's one of the handful that's in the ntp.org pools or another well known NTP master. Speaking as one of the 3841 servers in the pool.ntp.org pool, I'm happy to be described as a handful, something my mother used to say, but I do feel obligated to point out that it's a pretty big handful especially if you want to be fiddling ACLs on an hourly basis which is pretty much what it takes. I was thinking that the ntp.org servers on any particular network are a small set of exceptions to a general rule to rate limit outgoing NTP traffic. www.pool.ntp.org allows any NTP operator to opt-in to receive NTP traffic should their clock be available and accurate. - Jared
Re: TWC (AS11351) blocking all NTP?
I was thinking that the ntp.org servers on any particular network are a small set of exceptions to a general rule to rate limit outgoing NTP traffic. www.pool.ntp.org allows any NTP operator to opt-in to receive NTP traffic should their clock be available and accurate. I believe you, but I don't believe that the set of ntp.org servers changes so rapidly that it is beyond the ability of network operators to handle the ones on their own networks as a special case. Regards, John Levine, jo...@iecc.com, Primary Perpetrator of The Internet for Dummies, Please consider the environment before reading this e-mail. http://jl.ly
Re: TWC (AS11351) blocking all NTP?
I was thinking that the ntp.org servers on any particular network are a small set of exceptions to a general rule to rate limit outgoing NTP traffic. www.pool.ntp.org allows any NTP operator to opt-in to receive NTP traffic should their clock be available and accurate. I believe you, but I don't believe that the set of ntp.org servers changes so rapidly that it is beyond the ability of network operators to handle the ones on their own networks as a special case. There's a bootstrap issue here. I'm guessing that you may be picturing a scenario where a network operator simply queries to obtain the list of ntp.org servers and special-cases their own. However, I believe that the system won't add NTP servers that appear to be nonresponsive to the list (bootstrap paradox), and in any case the list of returned servers is quite large and a response basically picks a few random servers, so it is quite difficult to know what servers are on your network in an automated fashion. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
-48VDC supply for home lab?
Greetings NANOG'ers! I have a small home lab which I mostly use for learning and testing. I'm likely to receive some gear that needs negative 48VDC (ie: positive ground). Mains is a typical 120VAC, 60Hz. Can anyone recommend a power supply, reasonably priced, to go from 120VAC down to -48VDC@10Amps? Something that fits in a two post rack would be preferred, but not required. Thanks, Mark
Re: -48VDC supply for home lab?
On 2/3/2014 1:02 PM, Mark Leonard wrote: Greetings NANOG'ers! I have a small home lab which I mostly use for learning and testing. I'm likely to receive some gear that needs negative 48VDC (ie: positive ground). Mains is a typical 120VAC, 60Hz. Can anyone recommend a power supply, reasonably priced, to go from 120VAC down to -48VDC@10Amps? Something that fits in a two post rack would be preferred, but not required. Thanks, Mark Mark, I'd recommend a Kepco PRR 48-22M. We have one in-office, used it for some -48VDC equipment (Adtran Total Access gear) that we tested in-office. Worked great, and can be found on eBay for under $400 -Bobby
Re: TWC (AS11351) blocking all NTP?
On Mon, 03 Feb 2014 11:29:21 -0600, Joe Greco said: There's a bootstrap issue here. I'm guessing that you may be picturing a scenario where a network operator simply queries to obtain the list of ntp.org servers and special-cases their own. However, I believe that the system won't add NTP servers that appear to be nonresponsive to the list (bootstrap paradox), and in any case the list of returned servers is quite large and a response basically picks a few random servers, so it is quite difficult to know what servers are on your network in an automated fashion. And even harder to identify stuff that's downstream at one of your customer's sites. pgprFt4VVVKkV.pgp Description: PGP signature
Re: TWC (AS11351) blocking all NTP?
On 02/03/2014 12:50 PM, John R. Levine wrote: I was thinking that the ntp.org servers on any particular network are a small set of exceptions to a general rule to rate limit outgoing NTP traffic. www.pool.ntp.org allows any NTP operator to opt-in to receive NTP traffic should their clock be available and accurate. I believe you, but I don't believe that the set of ntp.org servers changes so rapidly that it is beyond the ability of network operators to handle the ones on their own networks as a special case. The list is large enough, and changes often enough, that filtering on it isn't likely to be successful. Also, the list of what are your servers can change without warning. Doug
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 12:38 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Mon, Feb 3, 2014 at 2:42 PM, Peter Phaal peter.ph...@gmail.com wrote: On Mon, Feb 3, 2014 at 10:16 AM, Christopher Morrow morrowc.li...@gmail.com wrote: On Mon, Feb 3, 2014 at 12:42 PM, Peter Phaal peter.ph...@gmail.com wrote: There's certainly the case that you could drop acls/something on equipment to selectively block the traffic that matters... I suspect in some cases the choice was: 50% of the edge box customers on this location are a problem, block it across the board here instead of X00 times (see concern about tcam/etc problems) I agree that managing limited TCAM space is critical to the scaleability of any mitigation solution. However, tying up TCAM space on every edge device with filters to prevent each new threat is likely yup, there's a tradeoff, today it's being made one way, tomorrow perhaps a different way. My point was that today the percentage of sdn capable devices is small enough that you still need a decimal point to measure it. (I bet, based on total devices deployed) The percentage of oss backend work done to do what you want is likely smaller... the folk in NZ-land (Citylink, reannz ... others - find josh baily / cardigan) are making some strides, but only in the exchange areas so far. fun stuff... but not the deployed gear as an L2/L3 device in TWC/Comcast/Verizon. I agree that today most networks aren't SDN ready, but there are inexpensive switches on the market that can perform these functions and for providers that have them in their network, this is an option today. In some environments, it could also make sense to drop in a layer switches to monitor and control traffic entering / exiting the network. The current 10G upgrade cycle provides an opportunity to deploy by 'current 10g upgrade cycle' you mean the one that happened 2-5 yrs ago? or somethign newer? did you mean 100G? I was referring to the current upgrade cycle in data centers, with servers connected with 10G rather than 1G adapters. The high volumes are driving down the cost of 10/40/100G switches. equipment that is SDN capable. The functionality required for this use case is supported by current generation merchant silicon and is widely available right now in inexpensive switches. right... and everyone is removing their vendor supported gear and replacing it with pica8 boxes? The reality is that as speeds/feeds have increased over the last while basic operations techiques really haven't. Should they? maybe? will they? probably? is that going to happen on a dime? nope. Again, I suspect you'll see smaller deployments of sdn-like stuff 'soon' and larger deployments when people are more comfortable with the operations/failure modes that change. Not just Pica8, most vendors (branded or white box) are using the same Broadcom merchant silicon, including Cisco, Juniper, Arista, Dell/Force10, Extreme etc.: http://blog.sflow.com/2014/01/drivers-for-growth.html Specifically looking at sFlow, large flood attacks can be detected within a second. The following article describes a simple example using integrated hybrid OpenFlow in a 10/40G ToR switch: hopefully there's some clamp on how much change per device/port you plan too? :) I'd hate to see the RP/RE/etc get so busy programming tcam that bgp/isis/ospf/etc flaps :( With integrated hybrid OpenFlow, there is very little activity on the OpenFlow control plane. The normal BGP, ECMP, LAG, etc. control planes handles forwarding of packets. OpenFlow is only used to selectively override specific FIB entries. that didn't really answer the question :) if I have 10k customers behind the edge box and some of them NOW start being abused, then more later and that mix changes... if it changes a bunch because the attacker is really attackers. how fast do I change before I can't do normal ops anymore? Good point - the proposed solution is most effective for protecting customers that are targeted by DDoS attacks. While trying to prevent attackers entering the network is good citizenship, the value and effectiveness of the mitigation service increases as you get closer to the target of the attack. In this case there typically aren't very many targets and so a single rule filtering on destination IP address and protocol would typically be effective (and less disruptive to the victim that null routing). Typical networks probably only see a few DDoS attacks an hour at the most, so pushing a few rules an hour to mitigate them should have little impact on the switch control plane. based on what math did you get 'few per hour?' As an endpoint (focal point) or as a contributor? The problem that started this discussion was being a contributor...which I bet happens a lot more often than /few an hour/. I am sorry, I should have been clearer, the SDN solution I was describing is aimed at protecting the target's links, rather than mitigating
Re: TWC (AS11351) blocking all NTP?
On Mon, 03 Feb 2014 16:49:37 +1300 Geraint Jones gera...@koding.com wrote: We block all outbound UDP for our ~200,000 Users for this very reason (with the exception of some whitelisted NTP and DNS servers). So far we have had 0 complaints I've heard this sort of absence of complaint statement used to justify some sort of truth claim about how to operate a network a number of times before. There is a certain appeal to it, particularly in cases such as this and for certain types of networks and operators, but if nothing else, for those that do it, I would also like to see some additional analysis about what is being filtered. It leaves many unconvinced and left to conjecture what the right approach is otherwise. If you have done that analysis or if you could make available some of that data for a research project, it would be very helpful for everyone to see what the measurable effect is. It would also make for a useful research project. John
Re: TWC (AS11351) blocking all NTP?
On Mon, 3 Feb 2014 07:08:25 + Dobbins, Roland rdobb...@arbor.net wrote: There's nothing in IPv6 which makes any difference. The ultimate solution is antispoofing at the customer edge. There is at least one small thing that may change some part of this and similar problems. If the threat vector were only accessible on IPv6 and that service on those systems is not easily discoverable then it will probably reduce the total population of systems being abused. I do realize in practice there are ways to discover systems, but the change in address architecture could change things, not perfectly, but I'll venture to suggest noticeably in some not so difficult to imagine scenarios. John
Re: -48VDC supply for home lab?
Tellabs stuff seems to work reasonably well: I've got a model 8001 -48VDC PDU in my lab rack at home, although it only supplies @ 1A, it does a fine enough job for what I need. Have a look at the Tellabs PS-1478 or so, which should do 10A. They're not explicitly rackmountable, but look like they'd easily be adapted to do so. You can still find PDF copies of Tellabs documentation online, too, and I believe the PS-1478 should be floating ground like my 8001. Hint: they can be had cheaply on ebay, since this is a home project, after all. That's where I obtained my 8001. There is still one seller around on ebay selling them for *way less* than $400, used. -- Jonathan Towne On Mon, Feb 03, 2014 at 01:08:28PM -0800, Robert Glover scribbled: # On 2/3/2014 1:02 PM, Mark Leonard wrote: # Greetings NANOG'ers! # # I have a small home lab which I mostly use for learning and testing. I'm # likely to receive some gear that needs negative 48VDC (ie: positive # ground). Mains is a typical 120VAC, 60Hz. # # Can anyone recommend a power supply, reasonably priced, to go from 120VAC # down to -48VDC@10Amps? Something that fits in a two post rack would be # preferred, but not required. # # Thanks, # Mark # # # Mark, # # I'd recommend a Kepco PRR 48-22M. We have one in-office, used it for # some -48VDC equipment (Adtran Total Access gear) that we tested # in-office. Worked great, and can be found on eBay for under $400 # # -Bobby #
Re: TWC (AS11351) blocking all NTP?
On Feb 4, 2014, at 5:38 AM, John Kristoff j...@cymru.com wrote: I do realize in practice there are ways to discover systems, but the change in address architecture could change things, not perfectly, but I'll venture to suggest noticeably in some not so difficult to imagine scenarios. I know you're aware of the work in this area, so I'll just note for the record that the 'IPv6 makes scanning impossible!' has already been exploded. ; --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton
Re: TWC (AS11351) blocking all NTP?
wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any is a good plan? I'd direct you at: https://www.nanog.org/resources/tutorials and particularly at: Tutorial: ISP Security - Real World Techniques II https://www.nanog.org/meetings/nanog23/presentations/greene.pdf On Mon, Feb 3, 2014 at 5:16 PM, Peter Phaal peter.ph...@gmail.com wrote: On Mon, Feb 3, 2014 at 12:38 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Mon, Feb 3, 2014 at 2:42 PM, Peter Phaal peter.ph...@gmail.com wrote: On Mon, Feb 3, 2014 at 10:16 AM, Christopher Morrow morrowc.li...@gmail.com wrote: On Mon, Feb 3, 2014 at 12:42 PM, Peter Phaal peter.ph...@gmail.com wrote: There's certainly the case that you could drop acls/something on equipment to selectively block the traffic that matters... I suspect in some cases the choice was: 50% of the edge box customers on this location are a problem, block it across the board here instead of X00 times (see concern about tcam/etc problems) I agree that managing limited TCAM space is critical to the scaleability of any mitigation solution. However, tying up TCAM space on every edge device with filters to prevent each new threat is likely yup, there's a tradeoff, today it's being made one way, tomorrow perhaps a different way. My point was that today the percentage of sdn capable devices is small enough that you still need a decimal point to measure it. (I bet, based on total devices deployed) The percentage of oss backend work done to do what you want is likely smaller... the folk in NZ-land (Citylink, reannz ... others - find josh baily / cardigan) are making some strides, but only in the exchange areas so far. fun stuff... but not the deployed gear as an L2/L3 device in TWC/Comcast/Verizon. I agree that today most networks aren't SDN ready, but there are inexpensive switches on the market that can perform these functions and for providers that have them in their network, this is an option today. In some environments, it could also make sense to drop in a layer switches to monitor and control traffic entering / exiting the network. it's probably not a good plan to forklift your edge, for dos targets where all you really need is a 3 line acl. The current 10G upgrade cycle provides an opportunity to deploy by 'current 10g upgrade cycle' you mean the one that happened 2-5 yrs ago? or somethign newer? did you mean 100G? I was referring to the current upgrade cycle in data centers, with servers connected with 10G rather than 1G adapters. The high volumes are driving down the cost of 10/40/100G switches. again, lots of cost and churn for 3 lines of acl... I'm not sold. With integrated hybrid OpenFlow, there is very little activity on the OpenFlow control plane. The normal BGP, ECMP, LAG, etc. control planes handles forwarding of packets. OpenFlow is only used to selectively override specific FIB entries. that didn't really answer the question :) if I have 10k customers behind the edge box and some of them NOW start being abused, then more later and that mix changes... if it changes a bunch because the attacker is really attackers. how fast do I change before I can't do normal ops anymore? Good point - the proposed solution is most effective for protecting customers that are targeted by DDoS attacks. While trying to prevent Oh, so the 3 line acl is not an option? or (for a lot of customers a fine answer) null route? Some things have changed in the world of dos mitigation, but a bunch of the basics still apply. I do know that in the unfortunate event that your network is the transit or terminus of a dos attack at high volume you want to do the least configuration that'll satisfy the 2 parties involved (you and your customer)... doing a bunch of hardware replacement and/or sdn things when you can get the job done with some acls or routing changes is really going to be risky. attackers entering the network is good citizenship, the value and effectiveness of the mitigation service increases as you get closer to the target of the attack. In this case there typically aren't very many targets and so a single rule filtering on destination IP address and protocol would typically be effective (and less disruptive to the victim that null routing). Typical networks probably only see a few DDoS attacks an hour at the most, so pushing a few rules an hour to mitigate them should have little impact on the switch control plane. based on what math did you get 'few per hour?' As an endpoint (focal point) or as a contributor? The problem that started this discussion was being a contributor...which I bet happens a lot more often than /few an hour/. I am sorry, I should have been clearer, the SDN solution I was describing is
Re: -48VDC supply for home lab?
I am using this: http://www.newark.com/xp-power/jpm160ps48/psu-160w-48v-3-3a/dp/97K2572 Locally it is available here for about $50 USD as new. I found it in a shop selling electronics for disco - don't tell them you are doing networks, that info will multiply the price by 10 :-). Regards, Baldur On 3 February 2014 22:02, Mark Leonard m...@bernoullinetworks.com wrote: Greetings NANOG'ers! I have a small home lab which I mostly use for learning and testing. I'm likely to receive some gear that needs negative 48VDC (ie: positive ground). Mains is a typical 120VAC, 60Hz. Can anyone recommend a power supply, reasonably priced, to go from 120VAC down to -48VDC@10Amps? Something that fits in a two post rack would be preferred, but not required. Thanks, Mark
Re: -48VDC supply for home lab?
I use: http://www.mastechpowersupply.com/dc-power-supply/switching-power-supply/volteq-power-supply-hy5020ex-50v-20a-over-voltage-over-current-protection/prod_61.html The output is changable from positive to negative ground by moving the shorting bar to ground from the - output to the + side. If you need to be able to charge a 48v battery plant you'd want the 60v version instead, but it's more $. The 50v one works fine for benchtesting equipment, at least. -Will On Mon, Feb 03, 2014 at 02:02:11PM -0700, Mark Leonard wrote: Date: Mon, 3 Feb 2014 14:02:11 -0700 Subject: -48VDC supply for home lab? From: Mark Leonard m...@bernoullinetworks.com To: North American Network Operators' Group nanog@nanog.org Greetings NANOG'ers! I have a small home lab which I mostly use for learning and testing. I'm likely to receive some gear that needs negative 48VDC (ie: positive ground). Mains is a typical 120VAC, 60Hz. Can anyone recommend a power supply, reasonably priced, to go from 120VAC down to -48VDC@10Amps? Something that fits in a two post rack would be preferred, but not required. Thanks, Mark
Re: TWC (AS11351) blocking all NTP?
On 4 Feb 2014, at 9:28 am, Christopher Morrow morrowc.li...@gmail.com wrote: wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any Which just pushes NTP to some other port, making control harder. We’ve already pushed all ‘interesting' traffic to port 80 on TCP, which has made traffic control very expensive. Let’s not repeat that history. -- Glen Turner http://www.gdt.id.au/~gdt/
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 2:58 PM, Christopher Morrow morrowc.li...@gmail.com wrote: wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any is a good plan? I'd direct you at: https://www.nanog.org/resources/tutorials and particularly at: Tutorial: ISP Security - Real World Techniques II https://www.nanog.org/meetings/nanog23/presentations/greene.pdf Thanks for the links. Many SDN solutions can be replicated using manual processes (or are ways of automating currently manual processes). Programmatic APIs allows the speed and accuracy of the response to be increased and the solution to be delivered at scale and at lower cost. it's probably not a good plan to forklift your edge, for dos targets where all you really need is a 3 line acl. For many networks it doesn't need to be forklift upgrade - vendors are adding programmatic APIs to their existing products (OpenFlow, Arista eAPI, NETCONF, ALU Web Services ...) - so a firmware upgrade may be all that is required. I do think that there are operational advantages to using protocols like OpenFlow, I2RS, BGP FlowSpec for these soft controls since they allow the configuration to remain relatively static and they avoid problems of split control (for example, and operator makes a config change and saves, locking in a temporary control from the SDN system). I would argue that the more specific the ACL can be the less collateral damage. Built-in measurement allows for a more targeted response. Good point - the proposed solution is most effective for protecting customers that are targeted by DDoS attacks. While trying to prevent Oh, so the 3 line acl is not an option? or (for a lot of customers a fine answer) null route? Some things have changed in the world of dos mitigation, but a bunch of the basics still apply. I do know that in the unfortunate event that your network is the transit or terminus of a dos attack at high volume you want to do the least configuration that'll satisfy the 2 parties involved (you and your customer)... doing a bunch of hardware replacement and/or sdn things when you can get the job done with some acls or routing changes is really going to be risky. I think an automatic system using a programmatic API to install as narrowly scoped a filter as possible is the most conservative and least risky option. Manual processes are error prone, slow, and blunt instruments like a null route can cause collateral damage to services. Typical networks probably only see a few DDoS attacks an hour at the most, so pushing a few rules an hour to mitigate them should have little impact on the switch control plane. based on what math did you get 'few per hour?' As an endpoint (focal point) or as a contributor? The problem that started this discussion was being a contributor...which I bet happens a lot more often than /few an hour/. I am sorry, I should have been clearer, the SDN solution I was describing is aimed at protecting the target's links, rather than mitigating the botnet and amplification layers. and i'd say that today sdn is out of reach for most deployments, and that the simplest answer is already available. The number of attacks was from the perspective of DDoS targets and their service providers. If you are considering each participant in the attack the number goes up considerably. I bet roland has some good round-numbers on number of dos attacks per day... I bet it's higher than a few per hour globally, for the ones that get noticed. The few per hour number isn't a global statistic. This is the number that a large hosting data center might experience. The global number is much larger, but not very relevant to a specific provider looking to size a mitigation solution. note that the focus of the original thread was on the contributors. I think the target part of the problem has been solved since before the slides in the pdf link at the top... Do most service providers allow their customers to control ACLs in the upstream routers? Do they automatically monitor traffic and insert the filters themselves when there is an attack? I don't believe so - while the slides describe a solution, automation is needed to make available at large scale. you're getting pretty complicated for the target side: ip access-list 150 permit ip any any log (note this is basically taken verbatim from the slides) view logs, see the overwhelming majority are to hostX port Y proto Z... filter, done. you can do that in about 5 mins time, quicker if you care to rush a bit. An automated system can perform the analysis and apply the filter in a second with no human intervention. What if you have to manage thousands of customer links? This brings up an interesting point use case for an OpenFlow
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 03, 2014 at 03:50:03PM -0500, John R. Levine wrote: I believe you, but I don't believe that the set of ntp.org servers changes so rapidly that it is beyond the ability of network operators to handle the ones on their own networks as a special case. I think you'd be surprised. I have to say I've been shocked at how little most network operators appear to understand about how NTP actually works, and how little thought is going into the consequences of suggested filtering techniques. Has anyone considered the implications of a world where your customers cannot correlate timestamps on abuse reports because you decided you knew better than they did how, and which sources of time they would be allowed to use? NTP works best with a diverse set of peers. You know, outside your little bubble, or walled garden, or whatever people in this thread appear to be trying to build. I'm not sure what to call it, but it's definitely not the Internet. --msa
Re: BCP38 [Was: Re: TWC (AS11351) blocking all NTP?]
On Feb 3, 2014 10:23 AM, Paul Ferguson fergdawgs...@mykolab.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 2/2/2014 2:17 PM, Cb B wrote: And, i agree bcp38 would help but that was published 14 years ago. But what? Are you somehow implying that because BCP38 was ...published 14 years ago (RFC2267 was initially published in 1998, and it was subsequently replaced by RFC2827)? I hope not, because BCP38 filtering would still help stop spoofed traffic now perpetuating these attacks, 14 years after BCP38 was published, because spoofing is at the root of this problem (reflection/amplification attacks). This horse is not dead, and still deserves a lot of kicking. $.02, - - ferg (co-author of BCP38) I completely agree. My sphere of influence is bcp38 compliant. And, networks that fail to support some form of bcp38 are nothing short of negligent. That said, i spend too much time taking defensive action against ipv4 amp udp attacks. And wishing others would deploy bcp38 does not solve today's ddos attacks. CB - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLv3ocACgkQKJasdVTchbLhowEAuO9DSQiRswVeqpHSccHo060h cqmIB8XlaNkzEPQw1w0A/0G6cjvtWBiJfwWbWoTY7X3RRMHeN36RkYR+2TonyNBi =W2wU -END PGP SIGNATURE-
Re: TWC (AS11351) blocking all NTP?
On Mon, Feb 3, 2014 at 7:40 PM, Glen Turner g...@gdt.id.au wrote: On 4 Feb 2014, at 9:28 am, Christopher Morrow morrowc.li...@gmail.com wrote: wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any Which just pushes NTP to some other port, making control harder. We've already pushed all 'interesting' traffic to port 80 on TCP, which has made traffic control very expensive. Let's not repeat that history. I think in the case of 'oh crap, customer is getting 100gbps of ntp...' the above (a third party notes that the 2nd line is redundant) is a fine answer, till the flood abates. I wouldn't recommend wholesale blocking of anything across an ISP edge, but for the specific case paul was getting at: ntp reflection attack target is your customer ... it's going to solve the problem.
Re: TWC (AS11351) blocking all NTP?
-larry directly since I'm sure he's either tired of this, or already reading it via the nanog subscription. On Mon, Feb 3, 2014 at 7:54 PM, Peter Phaal peter.ph...@gmail.com wrote: On Mon, Feb 3, 2014 at 2:58 PM, Christopher Morrow morrowc.li...@gmail.com wrote: wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any is a good plan? I'd direct you at: https://www.nanog.org/resources/tutorials and particularly at: Tutorial: ISP Security - Real World Techniques II https://www.nanog.org/meetings/nanog23/presentations/greene.pdf Thanks for the links. Many SDN solutions can be replicated using you're sort of a broken record on this bit ... I don't think folk are (me in particular) knocking sdn things, in general. In the specific though: 1) you missed the point originally, stop marketing your blog pls. 2) you missed the point(s) about availability and realistic deployment of solutions in the near term manual processes (or are ways of automating currently manual processes). Programmatic APIs allows the speed and accuracy of the response to be increased and the solution to be delivered at scale and at lower cost. and all of these require very strict and very careful deployment of oss measures to watch over current state and intended state. They require also very careful training and troubleshooting steps for the ops folk running the systems. None of this is deployable 'tomorrow' (in under 24hrs) safely, and most likely it'll be a bit more time until there is ubiquitous deployment of sdn-like functionality in larger scale networks. not that I'm not a fan, and not that I don't like me some automation, but.. having seen automation go very wrong (l3's acl spider... crushes l3..., flowspec 'whoopsie' at cloudflare and TWTC... there are lots of other examples). it's probably not a good plan to forklift your edge, for dos targets where all you really need is a 3 line acl. For many networks it doesn't need to be forklift upgrade - vendors are adding programmatic APIs to their existing products (OpenFlow, Arista eAPI, NETCONF, ALU Web Services ...) - so a firmware upgrade may be arista is deployed in which large scale networks with api/sdn functionality ? they're a great bunch of folks, they make some nice gear, it's still getting baked though, and it's not displacing (today) existing gear that's still being depreciated. for anything to be workable in the near-term, the above examples just aren't going to work. note my many references to 5-7 yrs when deprecation cycles and next-replacement happens all that is required. I do think that there are operational advantages to using protocols like OpenFlow, I2RS, BGP FlowSpec for these soft controls since they allow the configuration to remain relatively static and they avoid problems of split control (for example, and operator makes a config change and saves, locking in a temporary control from the SDN system). automation, with protections, safety checks, assurances that the process won't break things in odd failure modes.. not to mention bug^H^H^Hfeature issues with gear, we're still a bit from large scale deployment. I would argue that the more specific the ACL can be the less collateral damage. Built-in measurement allows for a more targeted response. sure, I think roland and I at least have been saying the same thing. Good point - the proposed solution is most effective for protecting customers that are targeted by DDoS attacks. While trying to prevent Oh, so the 3 line acl is not an option? or (for a lot of customers a fine answer) null route? Some things have changed in the world of dos mitigation, but a bunch of the basics still apply. I do know that in the unfortunate event that your network is the transit or terminus of a dos attack at high volume you want to do the least configuration that'll satisfy the 2 parties involved (you and your customer)... doing a bunch of hardware replacement and/or sdn things when you can get the job done with some acls or routing changes is really going to be risky. I think an automatic system using a programmatic API to install as narrowly scoped a filter as possible is the most conservative and least risky option. Manual processes are error prone, slow, and blunt instruments like a null route can cause collateral damage to services. folk say this, but the customer very often explicitly asks for null routes. The thing being targetted is very often not 'revenue generating ecommerce site', and for providers where the default answer is 'everything is a null route', their customers ought to find a provider that thinks differently. Typical networks probably only see a few DDoS attacks an hour at the most, so pushing a few rules an hour to mitigate them should
Re: BCP38 [Was: Re: TWC (AS11351) blocking all NTP?]
- Original Message - From: Cb B cb.li...@gmail.com I completely agree. My sphere of influence is bcp38 compliant. And, networks that fail to support some form of bcp38 are nothing short of negligent. That said, i spend too much time taking defensive action against ipv4 amp udp attacks. And wishing others would deploy bcp38 does not solve today's ddos attacks. Nope. But providing a bigger, better tuned hammer to apply to people's heads may. So any contributions you can make to http://www.bcp38.info will be cheerfully accepted. :-) Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
looking for a tool...
Hello, I was wondering if anyone could point me in the direction of a tool capable of sniffing (or reading pcap files), and reporting on lan station thruput in terms of bits per second. Ideally I'd like to be able to generate a sorted report of the top users and top thruputs observed and so forth. The traffic is pppoe and I need to monitor it at a specific switchport where I can arrange span. Thank you.
Re: looking for a tool...
Similar discussion not long ago mentioned tcptrace.org dre -- Forwarded message -- From: Andre Gironda an...@operations.net Date: Mon, Feb 3, 2014 at 9:05 PM Subject: Re: Do network diagnostic tools need upgrade? To: nanog@nanog.org nanog@nanog.org Cc: Ammar Salih ammar.alsa...@gmail.com Oldies, but goodies: shaperprobe (1st), pchar (3rd), tcptrace.org, lft (4th), iftop, nsping (2nd), iperf, sjitter, pathneck (3rd) These are newer -- http://www.internet2.edu/products-services/performance-monitoring/performance-tools/(OWAMP, 2nd) -- http://paris-traceroute.net (4th) -- http://packetdrill.googlecode.com dre On Mon, Feb 3, 2014 at 4:33 PM, Ammar Salih ammar.alsa...@gmail.com wrote: Hello NANOG list members, I have a question for you, are you happy with the current network diagnostic tools, like ping, trace route .. etc, don't you think it's time to have an upgraded version of icmp protocol? from my side there is a lot that I can NOT do with current tools and protocols, here are few scenarios, and I would like to hear yours: First scenario: To be able to troubleshoot advanced networks with complex QoS and policy-based routing configuration, where ping, traceroute and other network diagnostic tools do not provide accurate readings, for example, you are troubleshooting a web server with ping, and it looks functioning quite well (packet loss and round trip time is all good), but web services are still significantly slow, the fact is icmp and tcp:80 might have different priorities and bandwidth limits on each router along the path between the client and the server, in this case, network admins usually use telnet applications like (Paping), well it may help if the forward and return path of all packets are exactly the same. Second scenario: So another possible scenario is that you need to determine readings for forward and return paths, TraceRoute for example gives you forward path only using icmp. But what if you need to troubleshoot a VoIP server for example, assuming that packets return path might not be the same as forward path. Third scenario: One of the most common problems in networking, is that you don't have access to all equipment between client and server, but you have to troubleshoot the path between them and to understand where the problem exactly is in order to contact the right person without having the privilege to check the configuration on each router. Fourth scenario: Also, with trace route you can't determine the actual path, for example, the router may direct http traffic to proxy server while leaving other traffic going through a different hop. On Tue, Feb 4, 2014 at 8:34 AM, Mike mike-na...@tiedyenetworks.com wrote: Hello, I was wondering if anyone could point me in the direction of a tool capable of sniffing (or reading pcap files), and reporting on lan station thruput in terms of bits per second. Ideally I'd like to be able to generate a sorted report of the top users and top thruputs observed and so forth. The traffic is pppoe and I need to monitor it at a specific switchport where I can arrange span. Thank you.
Re: TWC (AS11351) blocking all NTP?
- Original Message - From: Glen Turner g...@gdt.id.au On 4 Feb 2014, at 9:28 am, Christopher Morrow morrowc.li...@gmail.com wrote: wait, so the whole of the thread is about stopping participants in the attack, and you're suggesting that removing/changing end-system switch/routing gear and doing something more complex than: deny udp any 123 any deny udp any 123 any 123 permit ip any any Which just pushes NTP to some other port, making control harder. We’ve already pushed all ‘interesting' traffic to port 80 on TCP, which has made traffic control very expensive. Let’s not repeat that history. Those who do not understand the Internet are condemned to reinvent it. Poorly. -- after henry@utzoo, though he was talking about Unix, and I am generally looking at Tapatalk and talking about Usenet. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Re: TWC (AS11351) blocking all NTP?
On 02/03/2014 05:10 PM, Majdi S. Abbas wrote: NTP works best with a diverse set of peers. You know, outside your little bubble, or walled garden, or whatever people in this thread appear to be trying to build. I'm not sure what to call it, but it's definitely not the Internet. The Internet is increasingly becoming something we want someone else to implement so that we can exploit it. Doug
