Re: google.com outage?
On Monday 09 May 2005 6:46 am, Paul Vixie wrote: -- Chris Keladis [EMAIL PROTECTED] wrote: Just a guess, but perhaps with www.google.com returning NXDOMAIN the gethostby* functions tried variations and ended up resolving sites like www.google.com.net which inadvertantly sent people to the spoof site. Seems plausible to me, anyway. not to me. RFC 1535 is VERY widely deployed, perhaps even universally so. I doubt it is universally so, as I think old versions of Windows did something similar in the 4.8.3 resolver, and they are still out there. Software takes a long time to die. However a lot of people saw the search engine sogo because their browser asked for www.google.com.net when it found www.Google.com didn't exist, and this is caught by a wildcard (allegedly). Although my quick inspection suggests that the DNS for these domains is very badly mangled, BIND 9 does somehow manage to get an IP address out of the mess it finds.
Google Web Accelerator
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi there, Did anyone catch this? Has anyone experienced any issues and if so, what steps did you take to address this? http://google.blognewschannel.com/index.php/archives/2005/05/05/much-controversy-over-googles-accelerator/ http://consumingexperience.blogspot.com/2005/05/google-web-accelerator-gwa-panacea-or_08.html http://www.searchenginejournal.com/index.php?p=1676 According to Google Blogoscoped (see below), the download page has been shut down because they can't handle the load. http://blog.outer-court.com/archive/2005-05-08-n20.html regards, /vicky -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCf6hJpbZvCIJx1bcRAsSiAKC1hRB4epeMef3FAxeC9/dSbfju9gCfSASO OUOZb1US1CLLZ8w/W5n1lnc= =v32F -END PGP SIGNATURE-
The evolution of NANOG
For some time there have been extensive discussions, both in person and via email, concerning the future of NANOG. A dedicated group of individuals (including many of you) has assembled a detailed proposal for NANOG's evolution, a proposal which lays out a number of important ideas. Merit staff are very grateful for the well-informed and constructive work put forth by the NANOG community. We at Merit are in agreement with most of the proposed changes, although there are a few points which we feel require further discussion. In this message I would like to outline Merit's reaction to these proposals and propose some next steps for discussion via email and at the next NANOG meeting in Seattle. As you will see below, our intention is to move forward with these changes over the summer. The first change you will see is one of financial transparency. Beginning with the Seattle meeting, Merit will provide an update on NANOG finances and operations at each meeting. Look for this presentation at every future NANOG meeting; it will take place during the Sunday Special Community Meeting in Seattle. Information will also be posted on nanog.org. We support the formation of a Steering Committee which will provide overall guidance for NANOG operations, appoint the Program Committee, and appoint the mailing list administrator panel. The Steering Committee would consist of six members elected by the NANOG community and one appointed by Merit. We propose that an election process be developed and that the first Steering Committee be elected during the summer of 2005, to begin serving by the Fall NANOG meeting. We support the formation of a Program Committee appointed by the Steering Committee. Our experience leads us to believe that a larger committee will be better able to handle the heavy workload, so we suggest that the Program Committee consist of 16 members (plus one member appointed by Merit). A smooth transition from the current committee to a new one is critical to ensure the success of upcoming meetings. We propose that the current Program Committee continue as presently constituted through the Fall NANOG meeting (except that the three open positions would be filled through appointment by the current committee after an open nomination process). This will allow time for the Steering Committee to be elected and to appoint a new Program Committee. In order to provide continuity and preserve institutional memory of the programming process, half (or more, at the discretion of the Steering Committee) of the members of the first reconstituted Program Committee would be selected from among those current Program Committee members willing to continue service, and the remainder would be new appointments from the community at large. We propose that Steve Feldman continue his capable leadership as Program Committee chair for the first year to help the new committee get established. We recommend that a member of the Steering Committee serve ex officio on the Program Committee, and that the chair of the Program Committee serve ex officio on the Steering Committee. These liaison roles will help ensure that the activities of the two groups are coordinated. We also support the change to mailing list administration, a change already partially implemented. We propose three panel members appointed by the Steering Committee and one appointed by Merit. We seek further discussion on the subject of membership. We at Merit believe that NANOG's success is due in no small part to its informality, an informality which mirrors other technical activities related to the Internet. We believe that addition of a paid membership category will dilute the influence of those who make the effort and take the time to participate in NANOG's main activity, the meetings. We propose that eligibility to vote in NANOG elections be determined solely by registration for NANOG meetings. We look forward to discussing this question in Seattle. Finally, although NANOG is a community of professionals, it is also a Merit business activity and not a separate organization. Merit accepts the risks, both financial and otherwise, created by NANOG. Merit staff are responsible first to Merit's Board of Directors for these financial and risk issues. In order to exercise our responsibility to our Board we must retain final authority over NANOG operations and finances, including representing NANOG to other organizations, negotiating and executing agreements with vendors, and selecting hosts and sites. We do not expect this to be a problem in practice, however, as we intend to work with the new Steering Committee and Program Committee on the presumption that the members of the NANOG community know best what they need from NANOG. We are confident that the goals of community guidance and transparency can be achieved without abrogating our responsibility to Merit's Board. You can read a draft of a proposed new NANOG Charter and action plan at
New UK Operators Forum
Hello, There has been a new Operators Forum setup in the UK. Please visit: http://www.ukif.org.uk/category.php?catid=44 For more details - we are hoping to open up the governance and have a strong UK operator focus within this group as the UK has lacked this. Regards, Neil
2005 Wiretap Report (CALEA)
2004 Wiretap report came out 4 May. http://www.govtech.net/magazine/channel_story.php?channel=3id=93846 Summary: 1700+ intercepts (punch list aggregate). The word VoIP or Internet does not appear anywhere in the 250+ page report. Gambling and Narcotics remain #1 reasons for a lawful order. Terrorism is usually handled FISA avenue, which are secret and not reported publicly. The official FBI document can be found here: http://www.askcalea.net/docs/2004wiretap.pdf -M -- Martin Hannigan (c) 617-388-2663 VeriSign, Inc. (w) 703-948-7018 Network Engineer IV Operations Infrastructure [EMAIL PROTECTED]
Peering BOF IX at NANOG in Seattle - The Great Public vs. Private Peering Debate
Hi all - Just wanted to invite you all to the upcoming Peering Birds-Of-a-Feather session at the upcoming NANOG, and give you a flavor of a couple of the topics to be discussed... Peering Introductions --- For Peering Coordinators who would lke to introduce themselves to the North American Peering Coordinator Community, we have some time set aside for you to introduce yourself and share with the Community: Company Name and Contact Information, AS#, Where you are peering or expect to peer in the future, What you are looking for in a peer, and Why people should be interested in peering with you. We have found these face-to-face interactions helps facilitate peering, particularly for folks coming in from overseas. It helps to bring network maps, and lots of business cards. If you email [EMAIL PROTECTED] this information I will make a slide with your contact information on it that will show up behind you are you speak. The Public vs. Private Peering Debate We have recruited two Peering Coordinators to articulate the two sides of the Great (Public vs. Private) Peering Debate. They have graciously agreed to share with the audience the Strongest Arguments for Public Peering (Maurice Dean), and the Strongest Arguments for Private Peering (Peter Cohen). Each side will have a few minutes to present their case, then a few minutes to attack the claims made by the other side and/or reinforce their own side of the argument as needed. We will perhaps add a few minutes in the middle for a couple of limited scope audience questions; those to help the speakers clarify a point (no speeches or attacks here). Both sides will then summarize their argument and the audience will be asked to vote for which side made the more compelling case. Audience Discussion --- As we did last Peering BOF, we will open the floor to discussion, focusing on points that one or both sides failed to make, or failed to make strongly enough, that would have perhaps made a difference in the audience vote. Background -- I researched this issue with a subset of the Peering Coordinator Community and shared the early results at the RIPE EIX WG meeting. If the discussions there are any indicator, I think we are in for an interesting and educational community discussion here. Below is an excerpt of the public vs. private peering arguments I heard from the Peering Coordinator Community and shared at the RIPE EIX WG meeting. I agree with the Peering Coordinators who believe the answer for most ISPs is a hybrid of public and private peering. I also agree that perhaps there sometimes emerges a transition based upon scale and strategic intent, but we will see what the community comes up with at the BOF. Bill PS - I cut and pasted the text below from the The Great (Public vs. Private Peering) Debate: Peering at 10G white paper that I am using to document these debates as they relate to 10 Gigabit-per-second Ethernet Peering. I am still looking for reviewers to provide feedback here BTW...If you are interested in this stuff and can spend a little time to provide feedback, send email to [EMAIL PROTECTED] When I feel more comfortable that I have it right, I will make the paper freely available to anyone who would like a copy. --- : snip : The Top 4 Reasons Public Peering is better than Private Peering 1. Aggregation Benefits a. A network can easily aggregate a large number of relatively small peering sessions across a single fixed-cost peering port, with zero incremental cost per peer. (Private peering requires additional cross connects and potentially an additional interface card, so there are real costs associated with each incremental peering session.) Small peering sessions often exhibit a high degree of variability in their traffic levels, making them perfect for aggregation. Since not all peers peak at the same time, multiple peers can be multiplexed onto the shared peering fabric, with one peers peak traffic filling in the valleys of another peer's traffic. This helps make peering very cost effective: I can't afford to dedicate a whole gigE card to private peering with this guy, but public peering is a no-brainer. b. Public peering ports usually have very large gradations of bandwidth: 100Mbps Ethernet upgrades to 1Gbps Ethernet, which upgrades to 10Gbps Ethernet. With such large gradations, it is easier for smaller peers to maintain several times more capacity via public peering than they are currently using, which reduces the likelihood of congestion due to shifting traffic patterns, bursty traffic, or uncontrolled Denial of Service attacks. Some peers aren't as responsive to upgrading their peering infrastructure, nor are they of similar mind with respect for the desire for peering bandwidth headroom[1]. The
DOS attack tracing
Hi, We recently experienced several DOS attacks which drove our backbone routers CPU to 100%. The routers are not under attack, but the router just couldn't handle the traffic. There is a plan to upgrade these routers. One criteria is the ability to track which IP address is under attack and blackhole the traffic quickly. Anyone can share your experience of what kind of router is capable of doing this? Also besides having a powerful router which can handle large volume of traffic, is there any other things that we need to consider in selecting the routers? Thanks, Richard
Re: DOS attack tracing
On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote: Hi, We recently experienced several DOS attacks which drove our backbone routers CPU to 100%. The routers are not under attack, but the router just couldn't handle the traffic. There is a plan to upgrade these routers. One criteria is the ability to track which IP address is under attack and blackhole the traffic quickly. Anyone can share your experience of what kind of router is capable of doing this? Also besides having a powerful router which can handle large volume of traffic, is there any other things that we need to consider in selecting the routers? I recently wrecked my car, totaling it and running down several small children on their way to sunday school in the process. I plan to upgrade my car, and one of the criteria is that it not crash and kill people. Can you share advice on what car is capable of doing this? This example is about as descriptive and useful at solving the problem as your original post. Without any details it is impossible to make any useful recommendation even if we wanted to. What type and scale of DoS are you trying to protect against, what type and scale of traffic are you routing, what kind of interfaces and how many, basic things like that. Without details, the best that you're likely to get (now that Dean is gone :P) is something akin to go buy a volvo, namely go buy a Juniper. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On the importance of IP address allocation....
Geoff Huston has done a great job of examining the issues surrounding the ITU-T proposal on IPv6 address allocation to national allocation registries. Definately worth a read. http://www.circleid.com/article.php?id=1078_0_1_0_C/ - ferg -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED] ferg's tech blog: http://fergdawg.blogspot.com/
Re: DOS attack tracing
On Mon, 9 May 2005, Richard wrote: : We recently experienced several DOS attacks which drove our backbone routers : CPU to 100%. The routers are not under attack, but the router just couldn't : handle the traffic. There is a plan to upgrade these routers. One criteria : is the ability to track which IP address is under attack and blackhole the : traffic quickly. Anyone can share your experience of what kind of router is : capable of doing this? : : Also besides having a powerful router which can handle large volume of : traffic, is there any other things that we need to consider in selecting the : routers? You shouldn't buy a bigger router just to handle DOS attacks. THere're many ways to address these types of issues using routers and/or servers. What is your normal CPU usage when there is no DOS attack? What does your capacity look like on the router interface where the DOS is coming in on? We need more info. scott
Re: DOS attack tracing
On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote: We recently experienced several DOS attacks which drove our backbone routers CPU to 100%. The routers are not under attack, but the router just couldn't handle the traffic. There is a plan to upgrade these routers. What kind of routers? We had problems like this with Cisco 7206VXRs with NPE-300s at my last job because they just couldn't handle the high volume of packets-per-second from certain types of attack. One criteria is the ability to track which IP address is under attack and blackhole the traffic quickly. Anyone can share your experience of what kind of router is capable of doing this? Disclaimer: I'm not an expert on this stuff, and it's possible (likely) that others on the list may have some other and / or better suggestions. Generally, I've seen this done by exporting flow data to another box, and then analyzing this data. I've used ehnt (extremely happy netflow tool) (http://ehnt.sourceforge.net/) to capture the flow data and export it to an easily machine-parsable feed, then used a Perl script to capture information on the top source / destination addresses. If there's interest, I could see whether it's possible to get this code and put it up somewhere (on an as-is basis) - the code was written by Kenytt Avery at Willing Minds (willingminds.com). We were keeping an ongoing log of such data, in case the router itself took a crap. On a Cisco router, you can also look at the raw cache flow data (sh ip cache flow), which has some summary data at the top, and then data on each flow. By rshing into the device and capturing this output, you have access to some other data to futz around with in some sort of script. So I'm not sure if there are any vendors which make it easy to figure this out while logged into the device itself (or whether this is a practical thing to do at all or something vendors are working on implementing), but it is possible to do using tools like netflow. w
RE: DOS attack tracing
On Mon, May 09, 2005 at 01:35:06PM -1000, Richard wrote: We recently experienced several DOS attacks which drove our backbone routers CPU to 100%. The routers are not under attack, but the router just couldn't handle the traffic. There is a plan to upgrade these routers. What kind of routers? We had problems like this with Cisco 7206VXRs with NPE-300s at my last job because they just couldn't handle the high volume of packets-per-second from certain types of attack. Oh... I guess that it would a known issue then... we have the exactly same type of routers. Our routers normally run at 35% CPU. What sucks is that the traffic volume doesn't have to be very high to bring down the router. On a Cisco router, you can also look at the raw cache flow data (sh ip cache flow), which has some summary data at the top, and then data on each flow. By rshing into the device and capturing this output, you have access to some other data to futz around with in some sort of script. So I'm not sure if there are any vendors which make it easy to figure this out while logged into the device itself (or whether this is a practical thing to do at all or something vendors are working on implementing), but it is possible to do using tools like netflow. So far we manually login to the router and use 'sh ip cache flow' on the router. It is ok, but not very effective. First when the router is slow to a halt, it is not even possible to the run the command most of the time. Secondly reading through the output and figuring out what's going on is not an easy task. I will definitely look into the tools to automate this process. Appreciate your suggestion. Just wonder if any router vendor has any built-in tools. Thanks, Richard
RE: DOS attack tracing
On Mon, 9 May 2005, Richard wrote: : We recently experienced several DOS attacks which drove our backbone : routers CPU to 100%. The routers are not under attack, but the : router just couldn't handle the traffic. There is a plan to upgrade : type of routers. Our routers normally run at 35% CPU. What sucks is that the : traffic volume doesn't have to be very high to bring down the router. That's because it's the number of packets per time period that it can't handle, not the traffic level. At this point it seems most likely that it's a simple UDP flood. If your CPU usually runs at 35% you definitely don't need a bigger router unless you're expecting a growth spurt. You might want to put an RRDTool or MRTG graph on the CPU usage to be sure. Depending on the size of your network you also might put a server at a good place where you can mirror the traffic to it and use NTop on the server. The software is free and will show a huge amount of detail if the server has the brawn to handle the load. More detail means more server brawn. You'll definitely see where the DOS is going. scott
Re: On the importance of IP address allocation....
I do wish circleid supported threading, or some other form of cross referencing Anyway most of those articles follow each other in quick succession. Look for all the articles by Paul Wilson, Geoff Huston and Tom Vest there. Yeah ok - the first few articles in their IP addressing and beyond category - http://www.circleid.com/channel/index/C0_8_1/ --srs On 5/10/05, Fergie (Paul Ferguson) [EMAIL PROTECTED] wrote: Geoff Huston has done a great job of examining the issues surrounding the ITU-T proposal on IPv6 address allocation to national allocation registries. Definately worth a read. http://www.circleid.com/article.php?id=1078_0_1_0_C/
RE: DOS attack tracing
On Mon, 9 May 2005, Scott Weeks wrote: On Mon, 9 May 2005, Richard wrote: : type of routers. Our routers normally run at 35% CPU. What sucks is that the : traffic volume doesn't have to be very high to bring down the router. That's because it's the number of packets per time period that it can't handle, not the traffic level. At this point it seems most likely that it's a simple UDP flood. If your CPU usually runs at 35% you definitely don't need a bigger router unless you're expecting a growth spurt. You might want to put an RRDTool or MRTG graph on the CPU usage to be sure. I'll disagree here. When you're engineering a network, what you generally need to care about is peak traffic, not average traffic. While DOS attack traffic is presumably traffic you'd rather not have, it tends to be part of the environment. This is somewhat of an arms race, and no router will protect you from all conceivable DOS attacks. That said, designing your network around the size of attack you typically see (plus some room for growth) raises the bar, and turns attacks of the size you've designed for into non-events that you don't need to wake up in the middle of the night for. Remember, the real goal in dealing with DOS attacks is to get to the point where you don't notice them, rather than just being able to explain why your network is down. For those attacks that go beyond the capacity you can afford, being able to divert the traffic is a good thing. The Riverhead system (now known as Cisco Guard, I think) does reasonably well at protecting networks downstream from it without being a big point of failure, but the network upstream from it still needs to be able to take the load. And being better able to characterize the attack traffic may help you ask your upstreams to block it for you. This can be done with some of the tools others have mentioned, including your router's flow cache *if your router hasn't already fallen over and died*. A rather dated paper on my experiences dealing with this sort of thing is at http://www.stevegibbard.com/ddos-talk.htm. -Steve
RE: DOS attack tracing
On Mon, 9 May 2005, Steve Gibbard wrote: : On Mon, 9 May 2005, Scott Weeks wrote: : On Mon, 9 May 2005, Richard wrote: : : : type of routers. Our routers normally run at 35% CPU. What sucks is that the : : traffic volume doesn't have to be very high to bring down the router. : : That's because it's the number of packets per time period that it can't : handle, not the traffic level. At this point it seems most likely that : it's a simple UDP flood. If your CPU usually runs at 35% you definitely : don't need a bigger router unless you're expecting a growth spurt. You : might want to put an RRDTool or MRTG graph on the CPU usage to be sure. : : I'll disagree here. Cool! Good 'ol operations discussion... :-) I took things out of order from your email, but kept the context. : www.stevegibbard.com/ddos-talk.htm Nice paper. However, you still say what I was saying, just in a different sort of way. Instead of NTop and RRDTool/MRTG, you use Cricket. RRDTool/MRTG alerts you to the problem and NTop directs you to the source of the problem. Once you get the procedure down pat, it can go pretty fast. As far as puttimg something in front of the core router(s) (such as Riverhead), I assumed there was nothing there for Richard; just raw router interface(s) to the upstream and not enough budget to afford those nice-but-expensive boxes. I was going to mention things like Riverhead or Packeteer later in the posts if appropriate. : When you're engineering a network, what you generally need to care about : is peak traffic, not average traffic. While DOS attack traffic is : presumably traffic you'd rather not have, it tends to be part of the : environment. : : This is somewhat of an arms race, and no router will protect you from all : conceivable DOS attacks. That said, designing your network around the : size of attack you typically see (plus some room for growth) raises the : bar, and turns attacks of the size you've designed for into non-events : that you don't need to wake up in the middle of the night for. This is what I was getting at. Engineering the network. That's more than buying a Bigger Badder Router and Fatter Pipes(BBRFP). If your router is running at 35% during the normal peak traffic flow, you don't need a BBRFP. All you need to do is design the network (and train the monkeys, as randy terms it... :-) to deal with extraordinary peaks. : Remember, the real goal in dealing with DOS attacks is to get to the point : where you don't notice them, rather than just being able to explain why : your network is down. Yes, but a BBRFP isn't the way to deal with this unless you've got the big budget. I know that a bigger hammer is better if you've got the money, but if you don't engineering finesse can work well. scott
NYT: Internet attack called broad and long lasting
Internet Attack Called Broad and Long Lasting by Investigators By JOHN MARKOFF and LOWELL BERGMAN Published: May 10, 2005 SAN FRANCISCO, May 9 - The incident seemed alarming enough: a breach of a Cisco Systems network in which an intruder seized programming instructions for many of the computers that control the flow of the Internet. [...] See the New York Times for the rest of the story.
Internet Attack Called Broad and Long Lasting by Investigators
SAN FRANCISCO, May 9 - The incident seemed alarming enough: a breach of a Cisco Systems network in which an intruder seized programming instructions for many of the computers that control the flow of the Internet. Now federal officials and computer security investigators have acknowledged that the Cisco break-in last year was only part of a more extensive operation - involving a single intruder or a small band, apparently based in Europe - in which thousands of computer systems were similarly penetrated. http://www.nytimes.com/2005/05/10/technology/10cisco.html?hpex=1115784000en=eeb27da2e75ec022ei=5094partner=homepage --Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb
