Re: effects of NYC power outage
[NANOG has been bouncing my attempts to reply to this thread for several days, possibly because I quoted the word u n st a b l e early on, apparently triggering the un subs cribe filter for words that start with uns and contain a b.. If your posts to NANOG have been silently bounced in the past, and your network's operational issues lead you to start your posts with words like uns tab le or uns ol vable or uns uit able or u nsp eak able, wonder no more. ] At any rate, about two days ago Senthil wrote: BGP was more [un st a ble] during code red propagation(http://www.renesys.com/projects/bgp_instability/.) A quick peek into both the graphs will make one thing clear: *BGP is robust enough to withstand any extreme congestion.* Anyone interested in this might also like to look at our report titled Internet Routing Behavior on 9/11 and in the Following Weeks. http://www.renesys.com/projects/reports/renesys-030502-NRC-911.pdf Note in particular the minute-by-minute changes in routing table size around critical events on pages 9 through 11. Fine time granularity is important to avoid missing all the interesting features. --jim
Re: effects of NYC power outage
On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge [EMAIL PROTECTED] wrote: Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th. Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : [EMAIL PROTECTED] http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/ I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data. Thanks! Craig Partridge Chief Scientist, BBN Technologies
Re: effects of NYC power outage
You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html BGP stability was normal on 9/11. As we know only the telephone network suffered more whereas internet remained stable. Their might have been some problems in the access because of the flash crowd problem. A particular slide from the nanog 23 presentation http://www.nanog.org/mtg-0110/ppt/misconfig/sld010.htm shows the behaviour on 9/11. Just observe closely the slide in the above link. It covers a period from a period from 8/1 to 9/26 and there was variation of 40-60 prefixes (between aug and september), except on 9/11 (there was 100 changes.) Only 0.1% of the route table was lost. BGP was more unstable during code red propagation(http://www.renesys.com/projects/bgp_instability/.) A quick peek into both the graphs will make one thing clear: *BGP is robust enough to withstand any extreme congestion.* But the question is: what can be an effective solution for access congestion on days like 9/11? __ Do You Yahoo!? Yahoo! Health - Feel better, live better http://health.yahoo.com
Re: effects of NYC power outage
In message [EMAIL PROTECTED], senthil ayyasa my writes: BGP stability was normal on 9/11. As we know only the telephone network suffered more whereas internet remained stable. Their might have been some problems in the access because of the flash crowd problem. I've now seen a lot of data on 9/11 and BGP (and other metrics) and, while final results and interpretation will wait for the NRC report, I will say that the data on reachability and such like varies dramatically, depending on where measured, granularity of measurements and other issues. Thanks! Craig
Re: effects of NYC power outage
Craig, We saw real hits on both Genuity and on NYC Teleglobe on Saturday. Both in latency and in packet loss. Our 9/11 graphs are visible at //order.mids.org/~peter/index.html where I put them following the event and on the NANOG 23 (Oct. 2001) site. Peter --- Peter H. Salus Chief Knowledge Officer, Matrix NetSystems Ste. 501W 1106 Clayton Lane Austin, TX 78723 +1 512 451-7602 ---
RE: effects of NYC power outage
A side-note on why 25 Broadway lost power. I am told they had the fuel, but the Local 3 union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry. --Phil -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; [EMAIL PROTECTED] Subject: Re: effects of NYC power outage On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge [EMAIL PROTECTED] wrote: Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th. Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : [EMAIL PROTECTED] http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/ I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data. Thanks! Craig Partridge Chief Scientist, BBN Technologies
RE: effects of NYC power outage
Nope. The main generator for the 5th floor apparently ran for a while, but the radiator became clogged with garbage floating aroung in the air, and therefore couldn't cool itself, and overheated. They shut it down to prevent it from hurting itself. Fuel was another issue. On Mon, 22 Jul 2002, Phil Rosenthal wrote: A side-note on why 25 Broadway lost power. I am told they had the fuel, but the Local 3 union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry. --Phil -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; [EMAIL PROTECTED] Subject: Re: effects of NYC power outage On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge [EMAIL PROTECTED] wrote: Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th. Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : [EMAIL PROTECTED] http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/ I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data. Thanks! Craig Partridge Chief Scientist, BBN Technologies -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
RE: effects of NYC power outage
I have never seen the final root cause (actually direct cause, we know what the root cause was) report from Telehouse. Although I can understand why Telehouse wouldn't want to say what happened. Between replacing water pumps, reports of contanimation inside and outside the cooling system, fuel delivery delays, etc I'm not certain there was a single cause. From the outside there seemed to be multiple events, each with different direct causes. On Mon, 22 Jul 2002, Alex Rubenstein wrote: Nope. The main generator for the 5th floor apparently ran for a while, but the radiator became clogged with garbage floating aroung in the air, and therefore couldn't cool itself, and overheated. They shut it down to prevent it from hurting itself. Fuel was another issue.
RE: effects of NYC power outage
From what I recall, it failed due to a mechanical problem first... then after they fixed it and had it running for sometime, it ran out of fuel. -Simon On Mon, 22 Jul 2002 11:50:29 -0400, Phil Rosenthal wrote: A side-note on why 25 Broadway lost power. I am told they had the fuel, but the Local 3 union worker who was watching the gauges on the generator misread the dials, and a human error caused the generator to run bone dry. --Phil -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Marshall Eubanks Sent: Monday, July 22, 2002 8:27 AM To: Craig Partridge; [EMAIL PROTECTED] Subject: Re: effects of NYC power outage On Mon, 22 Jul 2002 08:05:21 -0400 Craig Partridge [EMAIL PROTECTED] wrote: Anyone got good data comparing the effects on the Net (BGP reachability, etc) of this weekend's NYC power outage with the effects power outage late on September 11th. Hello; To be honest, I did not see any BGP or other routing effects from the NYC fire (there were problems on Abilene this weekend, but they were due to a bad router update). My data are presented on http://www.multicasttech.com/status and are fairly coarse-grained, having a 6 hour update cycle. The 9/11 problems actually came starting on 9/13 and 14 when the battery / generator power started running out at 25 Broadway. My understanding is that the biggest problem was the inability to access the facility to refuel. My (multicast-centric) analysis of the 9/11 response was presented at Nanog 23 : http://www.nanog.org/mtg-0110/eubanks.html You should also look at the other two presentations on 9/11 and the Internet at that meeting : http://www.nanog.org/mtg-0110/agenda.html Regards Marshall Eubanks T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : [EMAIL PROTECTED] http://www.multicasttech.com Test your network for multicast : http://www.multicasttech.com/mt/ I'm on a National Academy of Sciences committee looking at how the Internet fared on 9/11 and we're always in search of good comparative data. Thanks! Craig Partridge Chief Scientist, BBN Technologies
Re: effects of NYC power outage
I agree... I don't know why this is being discussed. I just thank -whoever- for 25bway still standing. -Simon On Mon, 22 Jul 2002 13:51:11 -0400 (EDT), Tuc wrote: From what I recall, it failed due to a mechanical problem first... then after they fixed it and had it running for sometime, it ran out of fuel. Hi, Ok, come on... That was 310 or so days ago. Exactly what happened shouldn't be a huge concern anymore. They addressed it, fixed it, and are making sure it doesn't happen again, thats the part we need to concentrate on. Phil, you weren't even a customer then, were you? And for those that didn't know/forgot I slept in the conference room the first 4 days after it all happened... And I think given the severity and magnitude of the event, they did a hell of a job. Only so much you can expect. Tuc/TTSG Internet Services, Inc.
Learning from the past (was Re: effects of NYC power outage)
Ok, come on... That was 310 or so days ago. Exactly what happened shouldn't be a huge concern anymore. They addressed it, fixed it, and are making sure it doesn't happen again, thats the part we need to concentrate on. The Morris worm happened over a decade ago. Computers are still being attacked using the same vulnerabilities used by the Morris worm, and amazingly some of the attacks are still working. The ATT New York City/FAA power failure happened over a decade ago (http://www.att.com/news/0991/910930.cha.html). Power problems continue to be a significant cause of network disruptions. ATT is a bit unusual. It almost always releases more information about its failures than any other telecommunications company. AS7007 happened over 5 years ago. Some networks still don't practicee safe filtering. Think volunteer fire department. The house you keep from burning down may be your own. If you don't want to participate, don't expect much help from your neighbors. Its amazing how often something happens to one organization, and continues to happen to other organizations. As an industry we want to make sure it not only doesn't happen to the same provider again, but the experience isn't repeated by other providers. That's why the electrical industry shares their experiences through DAWG (Disturbance Analysis Working Group) and the telephone industry shares their experiences through NRIC (Network reliability and interoperability council). I encourage folks to participate in the ISAC, NRIC and NSTAC programs. You may have the same vulnerability as several other providers, and don't know it. The solution you share may save yourself from a future vulnerability. The government cyber-protection groups have realized that they don't have good contacts with carrier hotel landlords, and it is an unknown exposure. Heck, there isn't even a good list of all the important carrier hotels. If you are a carrier hotel landlord, and aren't in contact with the government working groups examining infrastructure vulnerabilities, they want your input.
