And Verio too? (was Re: Level3 problems)
Authoritative sources report that Verio coincidentally had major problems last night also: http://www.boingboing.net/2005/10/21/two_tierone_isps_are.html http://slashdot.org/article.pl?sid=05/10/21/0958232 (is this the end for Level3? heh) Odd. The last time there was major instability due to multiple backbones upgrading was ... oh crap. So I'm going to get a major PSIRT upgrade now or die notice while I'm at NANOG. Good thing Monday evening is open... Pete. On Fri, 21 Oct 2005, Emilian Ursu wrote: I see its completely down and several others are starting to have problems. Anyone knows whats up ? Thanks
Methodology for BGP policy development
I'm looking for some good material on the methodology (best practices) of moderately-complex BGP policy development. I've found no shortage of the tools (prefix lists, community list filters, route maps, etc) for *implementation* of BGP policy. Including plenty of router configuration examples. I'm looking for help with the steps before the router configuration. What is a good methodology to go from a set of (~30-50) narrative descriptions (Propagate prefixes received from Customer Type X only to Peers Type Y) into a optimal, comprehensive set of community definitions, prefix/community/ASpath filters, route maps, peer templates, policy statements, etc? What methodology works for you? Are there presentations/papers/books/discussion threads that cover this aspect of routing policy development that you would recommend? Thanks for your help. Pete.
RE: optics pricing (Re: Weird GigE Media Converter Behavior)
On Sun, 29 Aug 2004, Michel Py wrote: 1. Support: sometimes you will need vendor support, and this is especially true of new products. Putting Kingston DRAM in a 2600 is one thing; a limited test on a few routers will quickly show if it works or not, and the odds of an IOS upgrade that would suddenly trigger the third-party memory to cause problems are close to zero, as DRAM as long passed the development stage and is now a commodity. OTOH, you won't have that many OC-192 IRs or LRs to play with. Maybe you'd try one third party PHY, then another one if the first one works, and so on. And suddenly something changes (which does happen with new products) and your vendor implements the changes on their PHYs but not on yours. You're screwed. I can see the vendor's concern about support. But it seems pretty hollow when after they lock me into a specific SFP, to help me, they mark it up 4x or more--because they can. If it was really about better customer experience, they would lock it down to an approved list of 3rd-party products, any of which could be purchased off the open market. Or they would publish a list of approved and/or supported 3rd-party optics, like Cisco used to do. Those customers who wanted to get the endorsed OEM product could buy those. And customers who wanted to cut corners at the risk of losing jobs and lower-quality service could do so. I've not seen any efforts by vendors to do anything about 3rd-party optics other than to prohibit them. So the vendors that still support 3rd-party optics must not be experiencing excruciating pain. As discussed at a recent NANOG, the vendor-specified modifications to the optics are trivial and do not justify the proprietary lock-up or the mark-up (if they did, then you'd expect the vendors to patent them and not have to lock them up). Unfortunately the only way this will change, if it can change, is with customer pressure, and to a very small extent, competitive pressure. Hopefully enough large vendors will allow 3rd-party optics so the threat to buy from the other guy will be credible. Pete.
Teaching/developing troubleshooting skills
I'm working on trying to teach others in my group (usually less-experienced, but not always) how to improve their large-network troubleshooting skills (the techniques of isolating a problem, etc). It's been so long since I learned network troubleshooting techniques I can't remember how I learned them or even how I used to do it (so poorly). Does anyone have experience with developing a skills-improvement program on this topic? If you've tried such a thing, what worked/didn't work for you? Outside training? Books? Mentoring? Motivational posters? I'm particularly sensitive to the I got my CCNA, therefore I know everything there is to know about troubleshooting perspective, and how to encourage improving troubleshooting skills without making it insultingly basic. Thanks for your help. Pete.
Re: TCP/BGP vulnerability - easier than you think
Interesting that Cisco uses random port selection with SNMP (http://www.cisco.com/warp/public/707/cisco-sa-20040420-snmp.shtml, see the Detail selection) but not with TCP. Too bad that TCP ports aren't randomized even with the fixed IOS versions. Would seem that as long as you're implementing security features like TCP RST confirmation, might as well implement randomized source ports. From Theo de Raadt at OpenBSD: http://archives.neohapsis.com/archives/openbsd/2004-04/1351.html This entire thing is being sold as `cross-vendor problem'. Sure. Some vendors have a few small issues to solve in this area. Minor issues. For us, those issues are 1/5 smaller than they are for other vendors. Post-3.5, we have fixes which make the problem even smaller. But one vendor -- Cisco -- has an *UTTERLY GIGANTIC HUGE* issue in this regard, and as you can see, they have not yet made an announcement see.. You are being told lots of people have a problem. By not seperating out the various problems combined in their notice, or the impact of those problems, you are not being told the whole truth. --- Pete. On Wed, 21 Apr 2004, Todd Vierling wrote: Date: Wed, 21 Apr 2004 11:37:04 -0400 (EDT) From: Todd Vierling [EMAIL PROTECTED] To: David Luyer [EMAIL PROTECTED] Cc: 'Patrick W.Gilmore' [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: TCP/BGP vulnerability - easier than you think On Wed, 21 Apr 2004, David Luyer wrote: : You missed the (assuming the attacker can accurately guess both : ports) part. : A significant number of BGP sessions will be with a source : port of 11000, 11001 or 11002; BGP sessions are generally : quite stable and Cisco routers start the source port at : 11000. If true, *that* would be a security risk in Cisco's port selection algorithm. Many modern OS's do not do simple sequential allocation of ports, making this point invalid.
Re: Sobigf + BGP
On Wed, 27 Aug 2003 [EMAIL PROTECTED] wrote: We have seen that many people *posting* do not have the best of intentions; I can assure you that there are lurkers on Nanog (surprise, surprise) who are not nearly as naive and well-intentioned as J. O. would hope. In fact, I know that there are subscribers from various print media, various on-line media, and certainly some stunningly unpleasant characters that I run into on other lists. And after being /.ed several times, there are undoubtedly end-users, small enterprises, non-network folks from networking companies, and assorted other groups which don't fit the traditional network operator mold. Oh, and sales people... Case in point: http://slashdot.org/articles/03/08/27/0214238.shtml?tid=111tid=126 references http://www.merit.edu/mail.archives/nanog/msg12818.html For those few finding the NANOG archives for the first time with this /. link, I'm sure they'll take some time to poke around recent threads with interesting titles like Sobigf + BGP Pete.
RE: East Coast outage?
On Fri, 15 Aug 2003, Randy Bush wrote: i guess it would be amusing to read a power engineers' mailing list discussing how the internet should have been designed. Well, if the Internet ever has a major outage, they'll be entitled to share their opinions. Until then...
RE: Cisco vulnerability and dangerous filtering techniques
On Wed, 23 Jul 2003, McBurnett, Jim wrote: Quick solution to this bug, as well as any future bug(s) replace all routers with PCs running Zebra. That is good until Zebra get's a bug and then someone will say go to XYZ... Macintosh running Zebra. Macs are as powerful as supercomputers, and they are inherently secure. Plus, who wouldn't give up the CLI for a candy-based interface that smiles at you? Pete.
Re: Backbone Infrastructure and Secrecy
On Tue, 8 Jul 2003, Adam Kujawski wrote: Who, besides Sean, has maps like this? The state PUC? If so, is that information available to the public? Do you have to go thorugh a background check and/or sign an NDA? Or is it only the providers themselves that have the maps for this stuff? It sounds to me like the secret is more that 25 carriers all use the same fiber bundle, and told their customers otherwise (we have dual entrances to 123 Anystreet, on our own fiber). Is it really any secret where the telco hotels are (http://www.carrierhotels.com) or where the incubent's CO's are (your local account team will happily show you a map)? Yes, this stuff takes time to assemble. How long does it take to coordinate 4 simultaneous plane hijackings? This is yet another case of keeping incredibly useful information from the people who could most use it (I'm sure the financial industry really appreciated finding out how vulnerable they are) to defend themselves, and make their vendors and government accountable, while assuming that the Bad Guys are too stupid to figure out how to get the information themselves. So, instead, we will all continue to blindly buy redundant infrastructure that uses the same fiber bundles, because we don't have the information to make a more intelligent choice. Just makes it easier for a terrorist to do his job. Pete.
Thank you
Thank you to everyone who attended NANOG 28 in Salt Lake City. We enjoyed hosting the conference and hope you enjoyed your time in Salt Lake. See you in October. Pete.
DNS records for routers
Any passionate opinions about DNS record conventions for routers? Or recommendations? I'm not particularly concerned about device naming conventions (we have that down), I'm more interested in what makes sense for public-viewable DNS names (so I can put those beautiful fully-compliant names where people can see them). Some traces show individual interface names, some just show device names. Any particular reason to go one way or the other for PTR records (doing a single device name for every interface seems easier and less-likely to screw up to me)? What about A records? A matching one per PTR, or just one A per device? Or no A records in the public DNS? Would round-robin A records (an A record for every interface address, all using the same device name) break anything (like performance measurement tools or network management tools)? Thanks. Pete.
Re: Network monitoring/IDS rant - What's hot what's not?
On Wed, 26 Feb 2003, Christopher L. Morrow wrote: CA-Unicenter/OVW/Tivoli are not IDS systems... (traditionally) but they can normally monitor the heck out of 'decent' sized networks (less than 500 components was my last experience with OVW atleast, tivoli and CA we never got working correctly with less than 1 metric butt ton of LOE to keep it running) What are the options and recommendations for networks 500 components? Pete.
Re: Network monitoring/IDS rant - What's hot what's not?
On Tue, 25 Feb 2003, Christopher J. Wolff wrote: I'm rapidly coming to the conclusion that any software Computer Associates publishes is designed for the criminally insane. http://www.sltrib.com/2003/feb/02232003/business/31810.asp
ENUM/E.164 books
Anyone have recommendations on good books (or similar resources) on ENUM/E.164 for education, planning, design, implementation and/or operation? Pete.
Experience w/ QoS/app performance monitors
Would anyone who is running QoS/SLA/application performance monitors (ie BRIX Networks) be willing to share (on the list or privately) what their experience has been with those products and how they are used/useful in actual experience to engineer and operate networks. Pete.
Selfish Routing
http://www.scienceblog.com/community/article1018.html --- The Internet is 'fault-tolerant,' so there are always many routes a message can take. A packet of data traveling from New York to San Francisco might go by way of Chicago or Dallas, or might even hop from New York to Columbus to Miami to Omaha to Denver to San Francisco. Routers have many ways to decide. Sometimes they send out test packets and time them. Sometimes routers exchange information about the condition of the network in their vicinities. But if routers choose the route that looks the least congested, they are doing selfish routing. As soon as that route clogs up, the routers change their strategies and choose other, previously neglected routes. Roughgarden has a suggestion that wouldn't be expensive to implement. Before deciding which way to send information, he says, routers should consider not only which route seems the least congested, but also should take into account the effect that adding its own new messages will have on the route it has chosen. That would be, he says, 'just a bit altruistic' in that some routers would end up choosing routes that were not necessarily the fastest, but the average time for all users would decrease. --- This might be easier to understand if it was more technical, but I'm only aware of a lot of disabled features on my routers that are supposed to in theory do some of these things. Abstractions and analogies aside, is this really a problem, and is it really worth solving? Sounds like a lot of additional complexity for the supposed benefits. Pete.
Regional Exchange Peering Forum
As a follow-up to the IX Operator Panel today, a Web site and mailing list have been set up to focus and expand the interests of regional exchange points. The REP Forum is intended for anyone who is interested in discussion and development of regional exchanges. This includes operators, participants and anyone interested in exploring or creating a regional exchange point. The REP Forum hopes to collect and share the knowledge and experience that has been developed by people who have built regional exchanges. The mailing list will be used to discuss and expand that knowledge-base and share ideas between REP projects. The REP Forum will provide information to create a regional exchange, or to make your current REP project more valuable (and hopefully more successful). This will initially focus on North American regional exchanges, though if there is interest from other areas of the world we will find a way accomodate those. The REP Forum web site is at http://www.rep.net . The mailing list is available by sending subscribe to [EMAIL PROTECTED] If you know of regional exchang operators that might not subscribe to the NANOG mailing list, please forward this message to them, or send me their contact information. Thanks. Pete.
IP QoS case-studies
I've found there's no shortage of advice and theory about the viability of IP QoS (DiffServ) in a large wide-area (converged) network. I have not had much luck with finding documentation about experiences implementing and operating such a beast. Presumably that's yet another (silent) confirmation that It Doesn't Work or There's a Better/Easier Way. Nevertheless, I'd still like to find anyone who has tried (successfully or not) to converge (ie VoIP/H.323/data) a high-speed (~ 1Gb/s) IP network and use IP QoS for what it is sold to do. White paper/presentation references or off-line conversation would be appreciated. Pete.
Re: Scaled Back Cybersecuruty
On 14 Jan 2003, Vijay Gill wrote: Avi Freedman [EMAIL PROTECTED] writes: Perhaps the Feds (and maybe states) could use their purchasing power to effect change. Short of that, or regulation, the I don't see how the serious issues we have with the 'net will get resolved. People do. I've been beating this particular horse for a while now, and we are starting to deploy the capex hammer. I suggest others start to do the same. See my presentation at the eugene nanog. I can see how purchasing power may motivate a vendor (and maybe lots of individual vendors) to fix their own problems, develop better products, or be more responsive. I'm trying to envision an RFP that awards business to one or a few network operators, but requires that they interoperate effectively with other operators who don't win any of the business. I've only got a state-level purchasing perspective, but I don't see it happening at any level. Is spending really an effective hammer (or gun) to make people work together if they aren't otherwise motivated to? Behavior related to the '96 Telecom Act doesn't inspire confidence. Can technical solutions be an effective band-aid for a complex poli-socio-economic problem like this? Pete.
Re: Trends in network operator security
On Thu, 9 Jan 2003, Sean Donelan wrote: On Wed, 8 Jan 2003 [EMAIL PROTECTED] wrote: Arent these more the attack trends of tier-3 providers and not network operators. Maybe. I don't see too many tier-1 network operators attacking other tier-1 network operators. The trend I continue to see affecting network operators is customer security incidents, i.e. compromised end-user applications. Would be nice to see all tier-X service providers provide more (working) knobs and response teams to help their customers and peers track, diagnose and defend and protect themselves against security attacks. Pete. http://pete.kruckenberg.com/blog
Re: Scaled Back Cybersecuruty
On Tue, 7 Jan 2003 [EMAIL PROTECTED] wrote: This may be of interst: AP: Bush Expected to Sign Scaled Back Internet Security Plan One of the criticisms of the change relative to this group is that the previous stronger wording for the network operator industry was watered down. Instead of expecting/demanding/mandating that the industry collaborate on network security (creating an ISAC and other measures), the latest draft simply recommends that the industry consider these measures. Is there anything happening with collaborative security policy and remediation in the industry? Has any effort showed progress towards an effective ISAC or similar? Can networks realistically collaborate on security, or do the political and operational barriers not justify the effort? Pete.
DWDM interconnects
How common are DWDM interconnects between networks (carriers)? Is DWDM considered a reliable/scalable/operable carrier interconnection technology? Is multi-vendor DWDM (whether internal to the network or for carrier interconnection) practical or sensible, especially for carrier/network interconnection? Many vendors proclaim interoperability, but does that work in the real world? Pete.
Re: AOL Cogent
On Sat, 28 Dec 2002, Richard A Steenbergen wrote: Consider this example: If I buy 100Mbit of transit from AboveNet in IAD, odds are you're gonna peer off 75% of my traffic locally, without it ever having touched expensive longhaul circuits. If I buy 100Mbit of paid peering, odds are you're going to be burning longhaul circuits carrying most of it all over the world, plus the same longhaul carrying it all back to me. So are any ISPs pricing transit and/or paid-peering bandwidth (significantly) lower if purchased at an exchange point? Pete.
Re: PAIX
Wired covered several of these topics in their August issue. http://www.wired.com/wired/archive/10.08/korea.html The article points out several subtle, yet fundamental, changes that happen socially and psychologically once the broadband network is available everywhere, to virtually everyone, all the time. We have yet to experience this in the US. I suspect that when it happens, it will be much different than we expect it to be, technically and otherwise. We still have to remember that for all the hype about the Internet, the killer app is still email and instant messenging. The killer apps on Internet2 (video conferencing, digital libraries, media-rich collaboration), which give some indication of what the future killer app will be, seem to be equally mundane (but exciting at the same time). Pete. On Thu, 14 Nov 2002 [EMAIL PROTECTED] wrote: On Thu, 14 Nov 2002 10:22:09 -0500 David Diaz wrote: 2) There is a lack of a killer app requiring peering every 100 sq Km. I recommend some quality time with journals covering South Korea, broadband, online gaming and video rental.
Good quotes on importance of good network addressing
I'm doing a presentation and white paper to convince a bunch of people that network addressing is one of (if not the most) important aspect of network design and management. The goal is to convince them that we need a plan, which will probably require them to do some renumbering. I'm looking for a couple of good quotes to include in the presentation. Anyone got some good ones from some Internet/network luminaries? Something to confirm the importance of a generally boring topic. Thanks. Pete.
Security/operations roles/interactions
I have the opportunity to redefine some roles of our Network Security group and Operations group. More specifically, Operations want to be more involved in day-to-day security activities like incident management and security monitoring. The goal is to make both groups more effective and valuable to our customers (and each other). I realize that solution will be specific to our organization and how we do business. But I hope to have a head-start and hopefully avoid some mistakes. It would be helpful to hear from the list how these groups seem to work best together and what division of labor has worked best to leverage their particular strengths. Private responses welcomed, and I'd be happy to report back to the list if there is interest in the results. Pete.
Re: looking glass
We have heavily modified a version of the MRLG ( ftp://ftp.enterzone.net/looking-glass/ ) to provide controlled router access to a specific (mostly internal) audience. We have found that allowing people who normally have no router access, to have read-only access to some normally enable-only commands through a Web interface has been invaluable in delegating diagnostics and peer review. The major benefit of a Web-based interface is that we can control the commands, input parameters, output display, and usability much better than with a command line interface. For example, we allow show config, but we cover up any security-sensitive information (passwords, SNMP strings, TACACS keys, server IP addresses, etc) in the command output. The control is very flexible, allowing certain users to see only certain things, or be able to execute commands that other users can't, for example. We can embed HTML links in the output to related resources (Web-based help, graphs, related commands, etc). Everything is encrypted via SSH/SSL, and can be tracked for audit and security purposes. To see something similar to what we have done (and where we got the idea from), see the Internet2 Abilene Core Node Router Proxy at http://loadrunner.uits.iu.edu/%7Erouterproxy/abilene/ Source code for the I2 Proxy is available from http://tseg.uits.indiana.edu/dist Pete. On Thu, 18 Jul 2002, Scott Granados wrote: Date: Thu, 18 Jul 2002 12:00:38 -0700 (PDT) From: Scott Granados [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: looking glass What are people using for looking glass software. Is it just some simple perl code which grabs data from the router or is it more complex than that? Thanks Scott
Sprint multicast route list
I'm doing some analysis of who I might be able to reach via multicast through Sprint. Sadly, route-views multicast peering with Sprint is not working at the moment. I'd appreciate if someone could email me the output from show ip mbgp neighbor sprint peer received-routes or show ip mbgp from a Sprint multicast BGP session. Thanks. Pete.
Re: Sprint multicast route list
Thanks, I got it. And route-views will be fixed, too. On Thu, 11 Jul 2002, Pete Kruckenberg wrote: Date: Thu, 11 Jul 2002 12:53:37 -0600 (MDT) From: Pete Kruckenberg [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Sprint multicast route list I'm doing some analysis of who I might be able to reach via multicast through Sprint. Sadly, route-views multicast peering with Sprint is not working at the moment. I'd appreciate if someone could email me the output from show ip mbgp neighbor sprint peer received-routes or show ip mbgp from a Sprint multicast BGP session. Thanks. Pete.
Re: Arbor Networks DoS defense product
On Wed, 15 May 2002, Sean Donelan wrote: Telus has gone first, and announced it is using Arbor's products across its backbone network. http://www.eweek.com/article/0,3658,s=720a=26867,00.asp People have been trying the products for a while. Does Arbor Networks really have an answer to DoS, or does it still need a little longer in the oven. Have any large networks gathered statistics on how much traffic DDoS/DoS/DRDoS attacks consume on an average day? The attacks I have been able to detect represent around 10-15% of my traffic on an on-going basis. I'm curious about the business case for investing in DoS defense mechanisms. DoS traffic is boosting service provider revenues through increased customer bandwidth usage. So the investment in defense mechanisms like Arbor would have to replace or increase that revenue. Will these issues inhibit wide-spread implementation of DoS defenses? Pete.
Effective ways to deal with DDoS attacks?
There's been plenty of discussion about DDoS attacks, and my IDS system is darn good at identifying them. But what are effective methods for large service-provider networks (ie ones where a firewall at the front would not be possible) to deal with DDoS attacks? Current method of updating ACLs with the source and/or destination are slow and error-prone and hard to maintain (especially when the target of the attack is a site that users would like to access). A rather extensive survey of DDoS papers has not resulted in much on this topic. What processes and/or tools are large networks using to identify and limit the impact of DDoS attacks? Thanks. Pete.
Re: Effective ways to deal with DDoS attacks?
On Wed, 1 May 2002 [EMAIL PROTECTED] wrote: and then again, there has been much discussion on simple DoS attacks, where the term DDoS is erroneously used... I am very much not trying to imply that this is the case here, but it's important that the two be thoroughly distinguished from each other - they are totally different things to deal with. Sorry, I should have been more clear. My issue (currently) is not being the target of the DDoS attack, but being a (unwilling) participant. People outside our network are launching DDoS attacks (distributed SYN floods) against destinations outside our network, using about 8,000 Web server hosts on our network as reflectors. These are not zombies. They are secured, uncompromised Web servers. The attack spoofs the target address as the source, and one of our machines as a destination, port 80. Getting everyone to implement defenses (SYN cookies) on their Web servers is nearly impossible (most don't even have a defense--printers and routers with Web interfaces). SYN packet comes in, one of these machines responses with a RST to the source, which is actually the target of the attack. Unfortunately, the target is often a site that people would like to get to, as is the reflector, so permanent filters on the target or reflector create lots of complaints. We captured several seconds of the last DDoS and came up with over 700 participating hosts... Some of them probably appear to be from our network... Pete.
Re: Effective ways to deal with DDoS attacks?
On Thu, 2 May 2002, Richard A Steenbergen wrote: SYN packet comes in, one of these machines responses with a RST to the source, which is actually the target of the You have an interesting situation. I think rate limiting outbound RSTs would be the least offensive thing you could do, off the top of my head. What about just blocking out-going RSTs altogether from our borders? While this interferes with proper TCP functionality, would it actually interfere enough to cause noticeable problems? Would certainly be less of a burden on routers than rate-limiting. Pete.
The Myth of Five 9's Reliability (fwd)
From the Canarie news mailing list. I don't think I've ever experienced five 9's on any telco service, I have always assumed I must be the one customer experiencing down-time, and the aggregate was somehow five 9's. How is network reliability calculated to end up with five 9's? Pete. -- Forwarded message -- Date: Wed, 24 Apr 2002 10:08:18 -0400 (EDT) From: [EMAIL PROTECTED] Subject: [news] The Myth of Five 9's Reliability For more information on this item please visit the CANARIE CA*net 3 Optical Internet program web site at http://www.canet3.net/news/news.html --- [A good article on the truth about five 9's reliability. Some excerpts - BSA] http://www.bcr.com/forum Deep Six Five-Nines? For much of the 20th century, the U.S. enjoyed the best network money could buy; hands-down, it was the most modern, most ubiquitous and most reliable in the world. And one term--five-nines--came to symbolize the network's robustness, its high availability, its virtual indestructibility. When the goal of five-nines was set, the network was planned, designed and operated by a monopoly, which was guaranteed a return on whatever it invested. It was in the monopoly's interest to make the network as platinum-plated as possible. One of the key points is that five-nines has long been somewhat overrated. Five-nines is NOT an inherent capability of circuit-switched, TDM networks. It's a manmade concept, derived from a mathematical equation, which includes some things and leaves out others. It's critical to remember that when you run the performance numbers on ALL the items in a network--those that are included in the five-nines equation and those that aren't--you're probably going to wind up with a number less than 99.999 percent. A well-run network actually delivers something around 99.45 percent. The gap between the rhetoric of five-nines and actual network performance leads to the conclusion that five-nines may not be a realistic or even necessary goal.
