Re: Fiber cut in SF area
Ong Beng Hui wrote: The problem of been LoS is a big problem in metro as far as I know. You can't just put a pair of FSO gear without going to the building owner to talk about rights and cost. Not forgetting lighting protection and other stuff. Murphy, Brian S CTR USAF ACC 83 NOS/Det 4 wrote: I haven't seen any mention of the possible use of FSO (Free Space Optics) by the provider to restore some reasonable amount of connectivity during an outage due to a fiber cut. I would expect that having 2 or 3 pairs of FSO boxes to provide a reduced failover capacity in metro areas would be a reasonable measure to ensure service for extended physical (fiber break, cut, backhoe) outages - although not necessarily for power. Yes, it would take some time to roll them out and set them up, but less time than the crew working the splices, and the folks handling the FSO boxes should be different from the fiber splice truck roll crew. Note that a power outage would not allow microwave to be an effective remediation method either. Plus, FSO's use of lasers (vice microwaves) means no issues with spectrum (AFAIK). Granted, they have limited distance and require LoS, but using two or more pairs can probably handle the 80% situation in the metro (unless there is data to indicate otherwise). murph Based on my experience with operating FSOs as infrastructure some years ago, the major limiting factor for FSOs is weather. In good weather, they should work just fine even at quite long ranges, providing that there is no obstruction or source of heat shimmer in the path, and you have carefully aimed your link to avoid sun outages. Bad weather (rain, snow, sandstorms, fog) causes very high levels of attenuation, with particularly bad weather reducing effective range to a few hundred meters at most. When this happens, the effect is area-wide, with a typical rain cell being a few km in size, so adding extra FSO links for redundancy is useless. If you've got a local airport nearby, you should be able to get good historical data for the frequency and duration of such weather conditions from METAR visibility data. For long-term standby installations, you've got to watch out for building work and cranes, which can pop up unexpectedly. However, if the link is being used solely as a protection path for rare failures in otherwise reliable fiber, and the alternative is either no protection path or a prohibitively expensive protection path, this may be perfectly acceptable: quite long ranges can be achieved with around 95-99% availability in typical European climates. You should expect installing and aiming a couple of FSO links at one another to take about a day in practice, unless you have a crack team of mobile laser ninjas trained and in readiness at all times (although the USAF may have greater access to ninjas, compared to to the rest of us). There is still the matter of getting permission for physical access, safety approval, access to power and network connectivity to the vantage points you will need to install the FSOs on, which can take much longer unless you already have it pre-planned. For truly rapid temporary links, I've seen one major UK operator actually just manually grout fiber in place along a kerbside to cover a few hundred meters of (presumably) temporary fiber run. This is probably faster to install than FSOs, even if the lifespan of such a link might be measured in days before someone crunches the fiber. -- Neil
Re: Fiber cut in SF area
Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge
RE: Fiber cut in SF area
True enough Jorge, however, we need full-orbed perspective hereit's not merely beating a dead horse; as far as topic goes, it is purely edification in the nth degree, manner, fashion. This is the lingua franca of this forum, and those who chose to read it, or not. Not merely pointed dialogue or geek speaks for the consummate net head ideologue. After all, iron sharpens iron. Demagoguery gives rise to elitism. No demonization here. You're ok. :-) Cheerio, Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 We move the information that moves your world. -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 9:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge __ This inbound email has been scanned by the MessageLabs Email Security System. __ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
RE: Fiber cut in SF area
Apologies for continuing this thread, but -- I don't understand this preoccupation with early warning systems on access to said manhole. What's the point? There are two possibilities here: 1) Someone goes down there and breaks something. You *already* know when this happens, because of your normal link monitoring. 2) There's a false positive (i.e. nothing malicious is done). From where I stand, these seem like ways to spend money in order to increase the reporting noise. Or am I missing something? Irregardless, it would be wise to focus on the *common* causes of outages. The things that happen and cause customers pain every day, due to more mundane occurrances like backhoes. Regardless of whether it's a hacksaw or a backhoe that takes out a cable, the customer is still down. Simple economics seem to dictate that the most attention should be devoted to the problems where you get the most bang for your buck - i.e. not movie theatre plot scenarios that happen once in many blue moons when there are so many other, far too common (and yet mundane) causes of outages. - S -Original Message- From: Peter Beckman beck...@angryox.com Sent: Monday, April 13, 2009 11:19 To: Dylan Ebner dylan.eb...@crlmed.com Cc: nanog@nanog.org nanog@nanog.org Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. You can't keep people out, especially since these manholes and tunnels are designed FOR human access. But a better job can be done of monitoring and knowing what is going on in the tunnels and access points from a remote location. Cheap: light sensor + cell phone = knowing exactly when and where the amount of light in the tunnel changes. Detects unauthorized intrusions. Make sure to detect all visible and IR spectrum, should someone very determined use night vision and IR lights to disable the sensor. Mid-Range: Webcam + cell phone = SEEING what is going on plus everything above. High-end: Webcam + cell phone + wifi or wimax backup both watching the entrance and the tunnels. James Bond: Lasers. Active monitoring of each site makes sure each one is online. Pros: * Knowing immediately that there is a change in environment in your tunnels. * Knowing who or at least THAT something is in there * Being able to proactively mitigate attempts * Availability of Arduino, SIM card adapters, and sophisticated sensor and camera equipment at low cost Cons: * Cell provider outage or spectrum blocker removes live notifications * False positives are problematic and can lower monitoring thresholds * Initial expense of deployment of monitoring systems Farmers use tiny embedded devices on their farms to monitor moisture, rain, etc. in multiple locations to customize irrigation and to help avoid loss of crops. These devices communicate with themselves, eventually getting back to a main listening post which relays the information to the farmer's computers. Tiny, embedded, networked devices that monitor the environment in the tunnels that run our fiber to help avoid loss of critical communications services seems to be a good idea. Cheap, disposable devices that can communicate with each other as well as back to some HQ is a way to at least know about problems of access before they happen. No keys to lose, no technology keeping people out and causing repair problems. Some other things that could detect access problems: * Pressure sensors (maybe an open manhole causes a detectable change in air pressure in the tunnel) * Temperature sensors (placed near access points, detects welding and thermite use) * Audio monitor (can help determine if an alert is just a rat squealing or people talking -- could even be automated to detect certain types of noises) * IR (heat) motion detection, as long as giant rats/rodents aren't a problem * Humidity sensors (sell the data to weatherbug!) One last thought inspired by the guy who posted about pouring quick-set concrete in to slow repair. Get some heavy-duty bags, about 10 feet long and large enough to fill the space in the tunnel. More heavily secure the fiber runs directly around the access space, then inflate two bags on either side of the access point. Easily deflated, these devices also have an electronic device which can notify HQ that they are being deflated or the pressure inside is changing (indicating pushing or manipulation). That way you only need to put these bags at access points, not throughout the whole tunnel. Kinda low-tech, but could be effective. No keys needed, could be inflated/deflated
Re: Fiber cut in SF area
True enough Jorge, however, we need full-orbed perspective hereit's not merely beating a dead horse; as far as topic goes, it is purely edification in the nth degree, manner, fashion. This is the lingua franca of this forum, and those who chose to read it, or not. Not merely pointed dialogue or geek speaks for the consummate net head ideologue. After all, iron sharpens iron. Demagoguery gives rise to elitism. No demonization here. You're ok. :-) I know, I don't mind the dialogue but IMHO besides trying to define which is the best way to seal a manhole, I'd rather see a more constructive discussion from an operational perspective. I really doubt that the big guys who own the fibers will make a rational decision about how to build their networks reading NANOG when the underlaying problem is not just technical or operational. For example, based on the experience with this outage, what's was out, how many users were affected, how the network operator's community handled the issue, what information was available, what kind of communications we used, what we did wrong, what we did right. BTW, now I know where to get a good padlock for my shack :-) Cheers Jorge
RE: Fiber cut in SF area
Cool enough. :-) Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 We move the information that moves your world. -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 11:31 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area True enough Jorge, however, we need full-orbed perspective hereit's not merely beating a dead horse; as far as topic goes, it is purely edification in the nth degree, manner, fashion. This is the lingua franca of this forum, and those who chose to read it, or not. Not merely pointed dialogue or geek speaks for the consummate net head ideologue. After all, iron sharpens iron. Demagoguery gives rise to elitism. No demonization here. You're ok. :-) I know, I don't mind the dialogue but IMHO besides trying to define which is the best way to seal a manhole, I'd rather see a more constructive discussion from an operational perspective. I really doubt that the big guys who own the fibers will make a rational decision about how to build their networks reading NANOG when the underlaying problem is not just technical or operational. For example, based on the experience with this outage, what's was out, how many users were affected, how the network operator's community handled the issue, what information was available, what kind of communications we used, what we did wrong, what we did right. BTW, now I know where to get a good padlock for my shack :-) Cheers Jorge __ This inbound email has been scanned by the MessageLabs Email Security System. __ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
RE: Fiber cut in SF area
Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 11:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge
RE: Fiber cut in SF area
Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Deepak Jain [mailto:dee...@ai.net] Sent: Tuesday, April 14, 2009 4:36 PM To: Gino Villarini; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area I don't mean to jump in here and state the obvious, but wireless links are not a panacea. At least a few folks have presented that fiber grooming has affected their *region*. It's not difficult to imagine that wherever the head link side (or agg point) of these regional wireless networks is... probably coincides with a fiber network or other telecom POP. You are just moving where your last mile vulnerabilities are (slightly.. as you are picking up multiple power vulnerabilities, Line of Sight, and other things along the way). In the example of a tornado or other weather disturbance, wireless links are subject to fade just as much as any kind of aerial wired asset. Deepak Jain AiNET -Original Message- From: Gino Villarini [mailto:g...@aeronetpr.com] Sent: Tuesday, April 14, 2009 4:12 PM To: Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 11:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge
RE: Fiber cut in SF area
Wireless RF links have their drawbacks: 1. Current GHz Frequency technology places upper limit of 1 Gbps on point-to-point links, and distance at 1 Gbps is limited. Commercial GiGE radios are just now appearing, replacing 100 Mbps Ethernet and oc3 SONET radios. Telco use of wireless links to backup 10/40 GiGE fiber trunks in metropolitan areas is not scalable. 2. Wireless technology contains hardware plethora of nuts, bolts, cables, fasteners, custom-tuned crystals, dishes, passive/active reflectors, in addition to layer 1 tuning best performed by EE specializing in RF. 3. Relative to fiber optic technologies, there is a very small circle of RF companies that can install, tune, and maintain wireless links correctly and reliably. 4. Tower-climbing/working skills are essential. But, what is the state of diverse telco fiber paths such that this fiber cut was not transparent to users in such a key US metropolitan area? -Original Message- From: Gino Villarini [mailto:g...@aeronetpr.com] Sent: Tuesday, April 14, 2009 1:42 PM To: Deepak Jain; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Deepak Jain [mailto:dee...@ai.net] Sent: Tuesday, April 14, 2009 4:36 PM To: Gino Villarini; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area I don't mean to jump in here and state the obvious, but wireless links are not a panacea. At least a few folks have presented that fiber grooming has affected their *region*. It's not difficult to imagine that wherever the head link side (or agg point) of these regional wireless networks is... probably coincides with a fiber network or other telecom POP. You are just moving where your last mile vulnerabilities are (slightly.. as you are picking up multiple power vulnerabilities, Line of Sight, and other things along the way). In the example of a tornado or other weather disturbance, wireless links are subject to fade just as much as any kind of aerial wired asset. Deepak Jain AiNET -Original Message- From: Gino Villarini [mailto:g...@aeronetpr.com] Sent: Tuesday, April 14, 2009 4:12 PM To: Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 11:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge
RE: Fiber cut in SF area
My point is more toward end users that need redundant options ... Im yet to find a Mcdonalds, a Bank Branch or a ATM that needs a GigE circuit ... Fixed Wireless in the 512 kbps to 6 Mbps range... SF area is serviced by Covad Wireless division among others, every major US city is served by at least 1 or 2 reputable business class Wireless ISP's. Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Holmes,David A [mailto:dhol...@mwdh2o.com] Sent: Tuesday, April 14, 2009 6:12 PM To: Gino Villarini; Deepak Jain; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Wireless RF links have their drawbacks: 1. Current GHz Frequency technology places upper limit of 1 Gbps on point-to-point links, and distance at 1 Gbps is limited. Commercial GiGE radios are just now appearing, replacing 100 Mbps Ethernet and oc3 SONET radios. Telco use of wireless links to backup 10/40 GiGE fiber trunks in metropolitan areas is not scalable. 2. Wireless technology contains hardware plethora of nuts, bolts, cables, fasteners, custom-tuned crystals, dishes, passive/active reflectors, in addition to layer 1 tuning best performed by EE specializing in RF. 3. Relative to fiber optic technologies, there is a very small circle of RF companies that can install, tune, and maintain wireless links correctly and reliably. 4. Tower-climbing/working skills are essential. But, what is the state of diverse telco fiber paths such that this fiber cut was not transparent to users in such a key US metropolitan area? -Original Message- From: Gino Villarini [mailto:g...@aeronetpr.com] Sent: Tuesday, April 14, 2009 1:42 PM To: Deepak Jain; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Deepak Jain [mailto:dee...@ai.net] Sent: Tuesday, April 14, 2009 4:36 PM To: Gino Villarini; Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area I don't mean to jump in here and state the obvious, but wireless links are not a panacea. At least a few folks have presented that fiber grooming has affected their *region*. It's not difficult to imagine that wherever the head link side (or agg point) of these regional wireless networks is... probably coincides with a fiber network or other telecom POP. You are just moving where your last mile vulnerabilities are (slightly.. as you are picking up multiple power vulnerabilities, Line of Sight, and other things along the way). In the example of a tornado or other weather disturbance, wireless links are subject to fade just as much as any kind of aerial wired asset. Deepak Jain AiNET -Original Message- From: Gino Villarini [mailto:g...@aeronetpr.com] Sent: Tuesday, April 14, 2009 4:12 PM To: Jorge Amodio; nanog@nanog.org Subject: RE: Fiber cut in SF area Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 -Original Message- From: Jorge Amodio [mailto:jmamo...@gmail.com] Sent: Tuesday, April 14, 2009 11:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Earth is a single point of failure. On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge
Re: Fiber cut in SF area
Gino Villarini wrote: Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. I'm pretty sure the WISPs in the Santa Cruz and Gilroy/Morgan Hill areas were all also taken offline due to the fiber cut. (Roy, can you verify, for south county?) Anyone in those areas who relied on a WISP as a backup to their fiber/copper link found that their redundant system wasn't really redundant after all. You may want to check (verify) how your 2 main providers handle their backhaul. jc
Re: Fiber cut in SF area
Gino Villarini wrote: SF area is serviced by Covad Wireless division among others, every major US city is served by at least 1 or 2 reputable business class Wireless ISP's. AFAIK Covad Wireless is just last mile wireless, and the route your packets take quickly merges with the local fiber/copper. jc
Re: Fiber cut in SF area
I think this issue has been beat. We're dealing with an arcaic system and protection at the same time... Mark Jackson, CCIE 4736 Senior Network, Security and Voice Architect 858-705-1861 markcciejack...@gmail.com Sent from my iPhone Please excuse spelling errors On Apr 14, 2009, at 3:24 PM, JC Dill jcdill.li...@gmail.com wrote: Gino Villarini wrote: SF area is serviced by Covad Wireless division among others, every major US city is served by at least 1 or 2 reputable business class Wireless ISP's. AFAIK Covad Wireless is just last mile wireless, and the route your packets take quickly merges with the local fiber/copper. jc
Re: Fiber cut in SF area
Gino Villarini wrote: Here in my area most of business outfits that require maximum availability of Internet or WAN conenctions have implemented dual connections from dual providers, most with a fiber/copper main and a fixed wireless backup. This trend goes from banks to Mcdonalds Gino A. Villarini g...@aeronetpr.com Aeronet Wireless Broadband Corp. tel 787.273.4143 fax 787.273.4145 A large company in the affected area had a T3 supplied by ATT and a wireless link to another ISP that was fed by two metro-ethernet links by companies other than ATT. All three uplinks were lost. So much for having backups,
Re: Fiber cut in SF area
JC Dill wrote: Gino Villarini wrote: Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. I'm pretty sure the WISPs in the Santa Cruz and Gilroy/Morgan Hill areas were all also taken offline due to the fiber cut. (Roy, can you verify, for south county?) Anyone in those areas who relied on a WISP as a backup to their fiber/copper link found that their redundant system wasn't really redundant after all. You may want to check (verify) how your 2 main providers handle their backhaul. jc It based on where the WISP fiber feed was located but in general they were all down. There were some special edge cases that stayed up fed from distant mountain tops. It didn't seem to matter who your upstream ISP was, they were all gone.
Re: Fiber cut in SF area
Roy wrote: JC Dill wrote: I'm pretty sure the WISPs in the Santa Cruz and Gilroy/Morgan Hill areas were all also taken offline due to the fiber cut. (Roy, can you verify, for south county?) Anyone in those areas who relied on a WISP as a backup to their fiber/copper link found that their redundant system wasn't really redundant after all. It based on where the WISP fiber feed was located but in general they were all down. There were some special edge cases that stayed up fed from distant mountain tops. It didn't seem to matter who your upstream ISP was, they were all gone. The little residential wireless provider I use (http://surfnetc.com/) in Santa Cruz county stayed up the whole time. I was surprised. (Looks like their uplink is via pnap (Internap).) -- Tony Rall
Re: Fiber cut in SF area
I haven't seen any mention of the possible use of FSO (Free Space Optics) by the provider to restore some reasonable amount of connectivity during an outage due to a fiber cut. I would expect that having 2 or 3 pairs of FSO boxes to provide a reduced failover capacity in metro areas would be a reasonable measure to ensure service for extended physical (fiber break, cut, backhoe) outages - although not necessarily for power. Yes, it would take some time to roll them out and set them up, but less time than the crew working the splices, and the folks handling the FSO boxes should be different from the fiber splice truck roll crew. Note that a power outage would not allow microwave to be an effective remediation method either. Plus, FSO's use of lasers (vice microwaves) means no issues with spectrum (AFAIK). Granted, they have limited distance and require LoS, but using two or more pairs can probably handle the 80% situation in the metro (unless there is data to indicate otherwise). murph - Date: Tue, 14 Apr 2009 15:57:52 -0700 From: Roy r.engehau...@gmail.com Subject: Re: Fiber cut in SF area To: JC Dill jcdill.li...@gmail.com Cc: nanog@nanog.org Message-ID: 49e514f0.7000...@gmail.com Content-Type: text/plain; charset=ISO-8859-1 JC Dill wrote: Gino Villarini wrote: Good points, some variables are dependant on the network infrastructure of the wireless provider. Localy, the main 2 providers have a copper/fiber independent networks. I'm pretty sure the WISPs in the Santa Cruz and Gilroy/Morgan Hill areas were all also taken offline due to the fiber cut. (Roy, can you verify, for south county?) Anyone in those areas who relied on a WISP as a backup to their fiber/copper link found that their redundant system wasn't really redundant after all. You may want to check (verify) how your 2 main providers handle their backhaul. jc It based on where the WISP fiber feed was located but in general they were all down. There were some special edge cases that stayed up fed from distant mountain tops. It didn't seem to matter who your upstream ISP was, they were all gone.
Re: Fiber cut in SF area
The problem of been LoS is a big problem in metro as far as I know. You can't just put a pair of FSO gear without going to the building owner to talk about rights and cost. Not forgetting lighting protection and other stuff. Murphy, Brian S CTR USAF ACC 83 NOS/Det 4 wrote: I haven't seen any mention of the possible use of FSO (Free Space Optics) by the provider to restore some reasonable amount of connectivity during an outage due to a fiber cut. I would expect that having 2 or 3 pairs of FSO boxes to provide a reduced failover capacity in metro areas would be a reasonable measure to ensure service for extended physical (fiber break, cut, backhoe) outages - although not necessarily for power. Yes, it would take some time to roll them out and set them up, but less time than the crew working the splices, and the folks handling the FSO boxes should be different from the fiber splice truck roll crew. Note that a power outage would not allow microwave to be an effective remediation method either. Plus, FSO's use of lasers (vice microwaves) means no issues with spectrum (AFAIK). Granted, they have limited distance and require LoS, but using two or more pairs can probably handle the 80% situation in the metro (unless there is data to indicate otherwise). murph
Re: Fiber cut in SF area
Mike Lewinski wrote: Joe Greco wrote: Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. The accepted standard is that a system is secure iff you can disclose _all_ of the details of how the system works to an attacker _except_ the private key and they still cannot get in -- and that is true of most open-standard or open-source encryption/security products due to extensive peer review and iterative improvements. What security by obscurity refers to are systems so weak that their workings cannot be exposed because then the keys will not be needed, which is true of most closed-source systems. It does _not_ refer to keeping your private keys secret. Key management is considered to be an entirely different problem. If you do not keep your private keys secure, no security system will be able to help you. S -- Stephen Sprunk God does not play dice. --Albert Einstein CCIE #3723 God is an inveterate gambler, and He throws the K5SSSdice at every possible opportunity. --Stephen Hawking smime.p7s Description: S/MIME Cryptographic Signature
Re: Fiber cut in SF area
On Mon, 13 Apr 2009 09:18:04 -0500 Stephen Sprunk step...@sprunk.org wrote: Mike Lewinski wrote: Joe Greco wrote: Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. The accepted standard is that a system is secure iff you can disclose _all_ of the details of how the system works to an attacker _except_ the private key and they still cannot get in -- and that is true of most open-standard or open-source encryption/security products due to extensive peer review and iterative improvements. What security by obscurity refers to are systems so weak that their workings cannot be exposed because then the keys will not be needed, which is true of most closed-source systems. It does _not_ refer to keeping your private keys secret. Correct. Open source and open standards are (some) ways to achieve that goal. They're not the only ones, nor are they sufficient. (Consider WEP as a glaring example of a failure of a standards process.) On the other hand, I was once told by someone from NSA that they design all of their gear on the assumption that Serial #1 of any new crypto device is delivered to the Kremlin. This principle, as applied to cryptography, was set out by Kerckhoffs in 1883; see http://www.petitcolas.net/fabien/kerckhoffs/ for details. Key management is considered to be an entirely different problem. If you do not keep your private keys secure, no security system will be able to help you. Yes. One friend of mine likens insecurity to entropy: you can't destroy it, but you can move it around. For example, cryptography lets you trade the insecurity of the link for the insecurity of the key, on the assumption that you can more easily protect a few keys than many kilometers of wire/fiber/radio. --Steve Bellovin, http://www.cs.columbia.edu/~smb
RE: Fiber cut in SF area
One thing that is missing here is before we can define security we need to define the threat and the obstruction the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them. So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the Threat to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, where do we go from here? It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.eb...@crlmed.com www.consultingradiologists.com -Original Message- From: Joe Greco [mailto:jgr...@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Thanks, *excellent* example. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Of course, but I said if we accept that. It was a challenge for the previous poster. ;-) Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Dylan Ebner wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. http://swm.pp.se/bayarea.jpg This is the bay area as well... How long do you need to spend with a torch to cut thru that? A couple of minutes? There is absolutely no way you can stop a determined attacker, and it would increase cost a lot more than it's worth. Time is better spent stopping the few people who actually do these kinds of things, same way as it's not worth it for regular people to wear body armour all the time, just in case they might get shot, or have parachutes and emergency exits that work in mid-flight on commercial airliners. The various police agencies and the NTSB cost less in a cost/benefit analysis. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Fiber cut in SF area
It all comes down to money... It will cost them lots of it to get power and some type of readers installed to monitor manhole access... There has always been a lack of security on the telco side, this incident just brings it to light... In my town many of the verizon fios boxes are not locked and the wiring frame boxes for pots line neither.. Its all of a matter of how much cash they wanna throw at it... Sent on the Now Network� from my Sprint® BlackBerry -Original Message- From: Dylan Ebner dylan.eb...@crlmed.com Date: Mon, 13 Apr 2009 09:57:30 To: nanog@nanog.org Subject: RE: Fiber cut in SF area One thing that is missing here is before we can define security we need to define the threat and the obstruction the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them. So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the Threat to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, where do we go from here? It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.eb...@crlmed.com www.consultingradiologists.com -Original Message- From: Joe Greco [mailto:jgr...@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Thanks, *excellent* example. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Of course, but I said if we accept that. It was a challenge for the previous poster. ;-) Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh
Re: Fiber cut in SF area
On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not minimize it. Flag this under the whole obscurity category, but think about this - if you're a vandal itching to do something stupid, and you see a bunch of manhole covers and a couple of them have locks on them, which ones are you going to target? The ones with the locks, of course. Why? Because by the very existence of the locks, it implies there's something of considerable value beyond the lock. -Andy
Re: Fiber cut in SF area
On 4/13/09, Dylan Ebner dylan.eb...@crlmed.com wrote: My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. Actually, in many ways it's getting easier; now, you can sign an NDA with your fiber providers and get GIS data for the fiber runs which you can pop into Google Earth, and verify path separation along the entire run; you put notification requirements into the contract stipulating that the fiber provider *must* notify you and provide updated GIS data if the path must be physically moved, and the move deviates the path by more than 50 feet from the previous GIS data; and you put escape clauses into the contract in case the re-routing of the fiber unavoidably reduces or eliminates your physical run diversity from your other providers. In years past, trying to overlay physical map printouts to validate path separation was a nightmare. Now, standardized GIS data formats make it a breeze. protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. Sometimes a little bit of paranoia can help save your butt...or at least keep you out of the hot seat. Matt
Re: Fiber cut in SF area
I guess the next generation fiber networks will need to be installed with tunnel boring machines and just not surface anywhere except the endpoints :) After all, undersea cables get along just fine without convenient access along their length... On Mon, Apr 13, 2009 at 12:12 PM, Mikael Abrahamsson swm...@swm.pp.sewrote: On Mon, 13 Apr 2009, Dylan Ebner wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. http://swm.pp.se/bayarea.jpg This is the bay area as well... How long do you need to spend with a torch to cut thru that? A couple of minutes? There is absolutely no way you can stop a determined attacker, and it would increase cost a lot more than it's worth. Time is better spent stopping the few people who actually do these kinds of things, same way as it's not worth it for regular people to wear body armour all the time, just in case they might get shot, or have parachutes and emergency exits that work in mid-flight on commercial airliners. The various police agencies and the NTSB cost less in a cost/benefit analysis. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Fiber cut in SF area
Or skip the locks and fill the manholes with sand. Then provide the service folks those big suction trucks to remove the sand for servicing :) On Mon, Apr 13, 2009 at 12:28 PM, Andy Ringsmuth andyr...@inebraska.comwrote: On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not minimize it. Flag this under the whole obscurity category, but think about this - if you're a vandal itching to do something stupid, and you see a bunch of manhole covers and a couple of them have locks on them, which ones are you going to target? The ones with the locks, of course. Why? Because by the very existence of the locks, it implies there's something of considerable value beyond the lock. -Andy
Re: Fiber cut in SF area
On Mon, 13 Apr 2009, Dorn Hetzel wrote: I guess the next generation fiber networks will need to be installed with tunnel boring machines and just not surface anywhere except the endpoints :) After all, undersea cables get along just fine without convenient access along their length... Boat anchors and earthquakes do a pretty effective job of cutting submarine cables. jms
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. You can't keep people out, especially since these manholes and tunnels are designed FOR human access. But a better job can be done of monitoring and knowing what is going on in the tunnels and access points from a remote location. Cheap: light sensor + cell phone = knowing exactly when and where the amount of light in the tunnel changes. Detects unauthorized intrusions. Make sure to detect all visible and IR spectrum, should someone very determined use night vision and IR lights to disable the sensor. Mid-Range: Webcam + cell phone = SEEING what is going on plus everything above. High-end: Webcam + cell phone + wifi or wimax backup both watching the entrance and the tunnels. James Bond: Lasers. Active monitoring of each site makes sure each one is online. Pros: * Knowing immediately that there is a change in environment in your tunnels. * Knowing who or at least THAT something is in there * Being able to proactively mitigate attempts * Availability of Arduino, SIM card adapters, and sophisticated sensor and camera equipment at low cost Cons: * Cell provider outage or spectrum blocker removes live notifications * False positives are problematic and can lower monitoring thresholds * Initial expense of deployment of monitoring systems Farmers use tiny embedded devices on their farms to monitor moisture, rain, etc. in multiple locations to customize irrigation and to help avoid loss of crops. These devices communicate with themselves, eventually getting back to a main listening post which relays the information to the farmer's computers. Tiny, embedded, networked devices that monitor the environment in the tunnels that run our fiber to help avoid loss of critical communications services seems to be a good idea. Cheap, disposable devices that can communicate with each other as well as back to some HQ is a way to at least know about problems of access before they happen. No keys to lose, no technology keeping people out and causing repair problems. Some other things that could detect access problems: * Pressure sensors (maybe an open manhole causes a detectable change in air pressure in the tunnel) * Temperature sensors (placed near access points, detects welding and thermite use) * Audio monitor (can help determine if an alert is just a rat squealing or people talking -- could even be automated to detect certain types of noises) * IR (heat) motion detection, as long as giant rats/rodents aren't a problem * Humidity sensors (sell the data to weatherbug!) One last thought inspired by the guy who posted about pouring quick-set concrete in to slow repair. Get some heavy-duty bags, about 10 feet long and large enough to fill the space in the tunnel. More heavily secure the fiber runs directly around the access space, then inflate two bags on either side of the access point. Easily deflated, these devices also have an electronic device which can notify HQ that they are being deflated or the pressure inside is changing (indicating pushing or manipulation). That way you only need to put these bags at access points, not throughout the whole tunnel. Kinda low-tech, but could be effective. No keys needed, could be inflated/deflated quickly, and you still get notification back to a monitoring point. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: [OT] Re: Fiber cut in SF area
On Sun, Apr 12, 2009 at 03:37:00AM +, Paul Vixie wrote: as long as the west's ideological opponents want terror rather than panic, and also to inflict long term losses rather than short term losses, that's true. in this light you can hopefully understand why bollards to protect internet exchanges against truck bombs are not only penny wise pound foolish (since the manholes a half mile away won't be hardened or monitored or even Of the two physical disaster scenarios, i.e. catastrophic destruction of a peering point or multiple long-line break, which do you think is the less costly -- in both time and treasure -- to remedy? It is acknowledged that the result of either is loss of service, but which is the more survivable event? In light of this, where would you focus your finite mitigation efforts? locked) but also completely wrongheaded (since terrorists need publicity which means they need their victims to be fully able to communicate.) Do you realize that you're putting trust in the sane action of parties who conclude their reasoning process with destruction and murder? -- . ___ ___ . . ___ . \/ |\ |\ \ . _\_ /__ |-\ |-\ \__
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: Peter Beckman [mailto:beck...@angryox.com] wrote: Sent: Monday, April 13, 2009 11:19 AM To: Dylan Ebner Cc: nanog@nanog.org Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. $500,000 to get it built with off-the-shelf components, tested in hostile tunnel environments and functioning. Then $350 per device, which would cover 1000 feet of tunnel, or about $2000 per mile for the devices. I'm not sure how things are powered in the tunnels, so power may need to be run, or the system could run off sealed-gel batteries (easily replaced and cheap, powers device for a year), system can be extremely low power. Add a communication device ($1000) every mile or two (the devices communicate between themselves back to the nearest communications device). Total cost, assuming 3 year life span of the device, is about $3000 per mile for equipment, or $1000 per year for equipment, plus $500 per year per mile for maintenance (batteries, service contracts, etc). Assumes your existing cost of tunnel maintenance can also either replace devices or batteries or both. Add a speedy roomba like RC device in the tunnel with an HD cam and a 10 or 20 mile range between charging stations that can move to the location where an anomaly was detected, and save some money on the per-device cost. It could run on an overhead monorail, or just wheels, depending on the tunnel configuration and moisture content. Add yet another system -- an alarm of sorts -- that goes off upon any anomaly being detected, and goes off after 5 minutes of no detection, to thwart teenagers and people who don't know how sophisticated the monitoring system really is. Put the alarm half way between access points, so it is difficult to get to and disable. Network it all, so that it can be controlled and updated from a certain set of IPs, make sure all changes are authenticated using PKI or certificates, and now you've made it harder to hack. Bonus points -- get a communication device that posts updates via SSL to multiple pre-programmed or random Confickr-type domains to make sure the system continues to be able to communicate in the event of a large outage. Then amortize that out to our bills. Extra credit: would you pay for it? Assuming bills in the hundreds of thousands of dollars per month, maybe to the millions of dollars, and then figure out what an outage costs you according to the SLAs. Then figure out how much a breach and subsequent fiber cut costs you in SLA payouts or credits, multiply by 25%, and that's your budget. If the proposed system is less, why wouldn't you do it? The idea is inspired by the way Google does their datacenters -- use cheap, off-the-shelf hardware, network it together in smart ways, make it energy efficient, ... profit! Anyone want to invest? Maybe I should start the business. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
--- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... scott
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Scott Weeks wrote: --- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... My estimates are for getting something off the ground, equipment-wise, not operationally. What is the cost of the outages? And if this setup can detect un-reported backhoe activity via accelerometers BEFORE it slices through the cable and you can get someone out to investigate the activity before it gets cut, how much is that worth? And my estimate was for the hardware, not training, etc. I'm guessing existing NOCs can easily incorporate new SNMP traps or other methods of alerts into their system fairly easily. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
Peter Beckman [mailto:beck...@angryox.com] wrote: Sent: Monday, April 13, 2009 11:19 AM To: Dylan Ebner Cc: nanog@nanog.org Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? Then amortize that out to our bills. Extra credit: would you pay for it? Chris
RE: Fiber cut in SF area
On 4/13/2009 at 1:12 PM, Peter Beckman beck...@angryox.com wrote: On Mon, 13 Apr 2009, Scott Weeks wrote: --- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... My estimates are for getting something off the ground, equipment-wise, not operationally. What is the cost of the outages? But would alarms prevent any, or what proportion, of these incidents? From what we know of this specific one, would an alarm have stopped the perpetrator(s)? It would have bought the NOC five, ten minutes tops before they got the alarm on the circuit. And in practice would a manhole alarm translate to a call to Homeland Security to have the SEALs descend the site pronto, a police unit to roll by when it has the time, or is it going to be an ATT truck rolling by between calls? I'm guessing number two or three, probably three. So what would it get them in this case. If it doesn't deter these guys, who does it deter? And what are the costs of false alarms? What will the ratio of real alarms to false ones be? Maybe lower-stakes vandals take to popping the edge of manhole covers as a little prank. Or that one that triggers whenever a truck tire hits it right. Or the whole line of them that go off whenever the temperature drops below freezing. Or, what I am absolutely sure will happen, miscommunication between repair crews and the NOC about which ones are being moved or field crews opening them without warning the NOC (or even intra-NOC communication). Will they be a boy who cried wolf?
RE: Fiber cut in SF area
Hi Peter, You wrote: So, let's see. I'm pulling numbers out of my butt here, snip Total cost...is about $3000 per mile for equipment snip It could run on an overhead monorail snip Network it all snip Confickr-type domains to make sure I get the feeling you haven't deployed or operated large networks. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. Joel is right: the thing about the outdoors is there's a lot of it. The cost over time investment of copper and fiber communucations networks, power transmission networks, cable transmission networks is pretty well documented elsewhere. Google around a little for them. The investment is tremendous. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. Then amortize that out to our bills. Extra credit: would you pay for it? Assuming bills in the hundreds of thousands of dollars per month, maybe to the millions of dollars, and then figure out what an outage costs you according to the SLAs. Then figure out how much a breach and subsequent fiber cut costs you in SLA payouts or credits, multiply by 25%, and that's your budget. If the proposed system is less, why wouldn't you do it? SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi-path diversity. Here come the microwave towers! The idea is inspired by the way Google does their datacenters -- use cheap, off-the-shelf hardware, network it together in smart ways, make it energy efficient, ... profit! Works great inside four walls. Anyone want to invest? Maybe I should start the business. Nahh, I already have a web cam on my Smarties orb. What else do I really need? Chris
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: I get the feeling you haven't deployed or operated large networks. Nope. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. It depends on where you want to deploy it and how many miles you want to protect. I was thinking along the lines of $1.5 million for 1000 miles of tunnel, equipment only. It assumes existing maintenance crews would replace sensors that break or go offline, and that those expenses already exist. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. What if you had 5 minutes of advanced notice that something was happening in or near one of your Tunnels that served hundreds of thousands of people and businesses and critical infrastructure? Could you get someone on site to stop it? Maybe. Is it worth it? Maybe. Given my inexperience with large networks, maybe fiber cuts and outages due to vandals, backhoes and other physical disruptions are just what we hear about in the news, and that it isn't worth the expense to monitor for those outages. If so, my idea seems kind of silly. SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi-path diversity. Here come the microwave towers! *laugh* Thank goodness for standardized GIS data. :-) --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: [OT] Re: Fiber cut in SF area
I sense a thread moderation occurring here shortly. valdis.kletni...@vt.edu wrote: On Mon, 13 Apr 2009 14:39:23 EDT, Izaac said: Do you realize that you're putting trust in the sane action of parties who conclude their reasoning process with destruction and murder? And how is that different from a US general plotting destruction and the killing of enemy troops during an offensive? And yet we usually trust our generals and call them sane.
Re: Fiber cut in SF area
This all implies that the majority of fiber is in tunnels that can be monitored. In my experience, almost none of it is in tunnels. In NYC, it's usually buried in conduits directly under the street, with no access, except through the man holes which are located about every 500 feet. In LA, a large amount of the fiber is direct bored under the streets, with access from hand holes and splice boxes located in the grassy areas between the street and the side walks. Along train tracks, the fiber is buried in conduits which are direct buried in the direct along side the train tracks, with hand holes every 1000 feet or so. In any of these scenarios, especially in the third, where the fiber might run through a rural area with no road access and no cellphone coverage. Simply walk through the woods to the train tracks, put open a hand hole and snip, snip, snip, fiber cut. Shane Ronan On Apr 13, 2009, at 5:54 PM, Peter Beckman wrote: On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: I get the feeling you haven't deployed or operated large networks. Nope. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. It depends on where you want to deploy it and how many miles you want to protect. I was thinking along the lines of $1.5 million for 1000 miles of tunnel, equipment only. It assumes existing maintenance crews would replace sensors that break or go offline, and that those expenses already exist. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. What if you had 5 minutes of advanced notice that something was happening in or near one of your Tunnels that served hundreds of thousands of people and businesses and critical infrastructure? Could you get someone on site to stop it? Maybe. Is it worth it? Maybe. Given my inexperience with large networks, maybe fiber cuts and outages due to vandals, backhoes and other physical disruptions are just what we hear about in the news, and that it isn't worth the expense to monitor for those outages. If so, my idea seems kind of silly. SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi- path diversity. Here come the microwave towers! *laugh* Thank goodness for standardized GIS data. :-) --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
--- On Mon, 4/13/09, chris.ra...@nokia.com chris.ra...@nokia.com wrote: From: Peter Beckman Subject: RE: Fiber cut in SF area Total cost...is about $3000 per mile for equipment I get the feeling you haven't deployed or operated large networks. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. ATT: 888,000 route miles(1). Verizon: 485,000 route miles(2). If we assume that 1/4 of ATT and Verizon's route-miles are in the US(3), this would mean a capital expense of $666M and $364M respectively, not including any costs incurred for maintenance, monitoring, repair, false positive etc. In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. In the words of Randy Bush, I encourage my competitors to do this. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com 1) http://www.att.com/gen/press-room?pid=4800cdvn=newsnewsarticleid=26554 2) http://mediumbusiness.verizon.com/about/network.aspx 3) I believe this to be an underestimate.
Re: Fiber cut in SF area
On 14/04/2009, at 11:35 AM, David Barak wrote: In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. Some statistics about the effectiveness of car alarms and unmonitored house alarms would probably be useful here. Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. -- Nathan Ward
Re: Fiber cut in SF area
But that would not be NEBS Complient -PHB I have thought of air horns in my colo cage when a tech of mine messes up. --Original Message-- From: Nathan Ward To: nanog list Subject: Re: Fiber cut in SF area Sent: Apr 13, 2009 4:55 PM On 14/04/2009, at 11:35 AM, David Barak wrote: In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. Some statistics about the effectiveness of car alarms and unmonitored house alarms would probably be useful here. Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. -- Nathan Ward
Re: Fiber cut in SF area
Nathan Ward wrote: Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Most of these measures also presume no shared access. I don't know the layout in the area, but I would expect that some manholes/routes are shared usage and maintenance. Not that my rural self remembers what a manhole looks like under the lid. :) I'm betting inside job, which means redundant routes, security measures, etc all tend to go out the window unless some serious money goes into it, and even then, is there a security mechanism that can't be broken? Jack
Re: Fiber cut in SF area
There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P
RE: Fiber cut in SF area
Though I think networked environmental monitoring has its merits, it's clear the technology is unproven in monitoring fiber tunnels, and my inexperience in running and managing such tunnels makes this thread bordering on off-topic. I'm happy to continue conversations via email, but this will be my last on-list reply regarding the topic I started. On Mon, 13 Apr 2009, Crist Clark wrote: But would alarms prevent any, or what proportion, of these incidents? It's hard to say without researching. Sometimes such research shows amazing results that shock people in the industry. Hospitals were shocked to see surgical mistakes reduced by 80+% after implementing a checklist that both doctors and nurses had to go through prior to starting the procedure, and having the patient also go over and approve what was to be done. The stories you hear of people who are getting amputated writing this leg and X X X NOT THIS LEG before surgery is a result of these studies and checklists. RFID-tagged surgical components and gauze pads are another tech tool being used after such research. You'd think a checklist wouldn't really help, but in reality it made industry changing and life-saving differences. While active alarms and monitoring of fiber tunnels would do the same, but without research, nobody can say for sure how effective or ineffective such a system would be. From what we know of this specific one, would an alarm have stopped the perpetrator(s)? It would have bought the NOC five, ten minutes tops before they got the alarm on the circuit. And in practice would a manhole alarm translate to a call to Homeland Security to have the SEALs descend the site pronto, a police unit to roll by when it has the time, or is it going to be an ATT truck rolling by between calls? I'm guessing number two or three, probably three. So what would it get them in this case. If it doesn't deter these guys, who does it deter? It's not there as a deterrent. It's there to allow a NOC to know that something is going on in a tunnel where potentially critical infrastructure resides. Maybe it doesn't prevent the malicious cut, but combined with video surveilence, it could identify the cutters. Audio recording devices could record voices. I assume large networks have large 24/7 crews. Get a truck to roll (once you sufficiently trust the system) or get a contractor who resides nearby to check out the area. When the alarm goes off, you go check it. If you welded the manholes shut, and there are no scheduled maintenance windows for that area, you can be pretty damn sure something untoward is going on, or it'll be a company truck roll that didn't follow procedure. And what are the costs of false alarms? What will the ratio of real alarms to false ones be? Maybe lower-stakes vandals take to popping the edge of manhole covers as a little prank. Weld 'em shut. Use one of those special screws that you can only unscrew with the right equipment (worked wonders for the tire industry with the lock nut). It won't stop anyone determined, but 13 year olds with M80s will move on. If you get a certain location that continues to get false alarms due to vandals, put in a highpowered webcam to monitor the location. Use ZoneMinder to monitor and record motion. Make sure the camera does nighttime well. Then when you have an alarm, check the video. Or that one that triggers whenever a truck tire hits it right. I would envision that though every device would report the same data with the same sensitivity, false alarms could be mitigated through filters for a given location. Tunnels near train tracks would be filtered differently than tunnels in the middle of a field under high power lines. Or the whole line of them that go off whenever the temperature drops below freezing. The device would go through a lot of environmental testing, so that its upper and lower operating limits could be known. Hardened where necessary. Or, what I am absolutely sure will happen, miscommunication between repair crews and the NOC about which ones are being moved or field crews opening them without warning the NOC (or even intra-NOC communication). Will they be a boy who cried wolf? Maybe. Maybe the whole idea is way too far fetched. Maybe my impression of the state of affairs when it comes to fiber tunnels is really not that big of a deal, and that outages due to physical access (humans, backhoes, floods) don't make up a significant portion of outages, and this is not a problem that fiber companies want to solve. Clearly there are a lot of problems that this sort of monitoring could face. Given sufficient time to mature, I think cheap, repeatable monitoring devices networked together can be a valuable asset, rather than yet another annoying alarm NOC folk and maintenance crews grow to hate and simply not be effective. --- Peter
Re: Fiber cut in SF area
Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Better they cut the fiber instead of Oklahoma Citying the central office.
Re: Fiber cut in SF area
But you are ignoring the cost of designing, procuring, installing, monitoring, maintaining such a solution for the THOUSANDS of man holes and hand holes in even a small fiber network. The reality is, the types of outages that these things would protect against (intentional damage to the physical fiber) just don't happen often enough to warrant the cost. These types of solutions don't protect against back hoes digging up the fiber, as even if they gave a few minutes of advanced notice, the average telco can't get someone to respond to a site in an hour let alone minutes. On Apr 13, 2009, at 9:05 PM, Peter Beckman wrote: On Mon, 13 Apr 2009, Shane Ronan wrote: This all implies that the majority of fiber is in tunnels that can be monitored. In my experience, almost none of it is in tunnels. In NYC, it's usually buried in conduits directly under the street, with no access, except through the man holes which are located about every 500 feet. In LA, a large amount of the fiber is direct bored under the streets, with access from hand holes and splice boxes located in the grassy areas between the street and the side walks. Along train tracks, the fiber is buried in conduits which are direct buried in the direct along side the train tracks, with hand holes every 1000 feet or so. In any of these scenarios, especially in the third, where the fiber might run through a rural area with no road access and no cellphone coverage. Simply walk through the woods to the train tracks, put open a hand hole and snip, snip, snip, fiber cut. I'm sure more malicious fiber cuts would result in heightened security. If you can put your hand in it, you could put a sensor in it. It wouldn't work everywhere, but it could work even in conduit or just simply inside access points. A device the size of your fist or smaller could do the monitoring, and would fit in most access points I would guess. You can't protect it all, and obviously you can't put a camera at every access point (well, maybe you can). You can't stop a determined person from doing anything (like promote networked smart sensors for fiber runs, or setting a small explosion inside an access point). And maybe environmental monitoring of these areas just won't do anything to help. But who knows. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Fiber cut in SF area
On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach writes: protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. This completely ignores the grooming problem. Not completely; it just gives you teeth for exiting your contract earlier and finding a more responsible provider to go with who won't violate the terms of the contract and re-groom you without proper notification. I'll admit I'm somewhat simplifying the scenario, in that I also insist on no single point of failure, so even an entire site going dark doesn't completely knock out service; those who have been around since the early days will remember my email to NANOG about the gas main cut in Santa Clara that knocked a good chunk of the area's connectivity out, *not* because the fiber was damaged, but because the fire marshall insisted that all active electrical devices be powered off (including all UPSes) until the gas in the area had dissipated. Ever since then, I've just acknowledged you can't keep a single site always up and running; there *will* be events that require it to be powered down, and part of my planning process accounts for that, as much as possible, via BCP planning. Now, I'll be the first to admit it's a different game if you're providing last-mile access to single-homed customers. But sitting on the content provider side of the fence, it's entirely possible to build your infrastructure such that having 3 or more OC192s cut at random places has no impact on your ability to carry traffic and continue functioning. You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business. You simply count on them not playing entirely fair, and penalize them when they don't; and you have enough parallel contracts with different providers at different sites that outages don't take you completely offline.
Re: Fiber cut in SF area
On Apr 13, 2009, at 8:31 PM, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. I certainly think this trailer is the most insightful thought of the day. When you're looking for backup comms, is it just going to be the ham radio operators and am/fm radio stations left if there were some outage? With tv having gone digital it's not possible to tune in and pick up the audio carrier anymore. Wartime and times of civil unrest the first thing you do is take over communication to the citizens. Without your internet^Wpodcast of the news, how will you know what is going on? If redundancy is sacrificed in the name of better quarterly earnings is it the right decision? this is not only interesting from a network operators perspective but from a governance perspective as well. I've not done any ham radio stuff for ~15+ years but do keep a shortwave radio around (battery powered of course). The first thing to happen will be the network will be severed. Look at what happened in Burma. Both their internet links were turned off, and not just taking down BGP, but the circuits were unplugged. - jared
Re: Fiber cut in SF area
There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. I certainly think this trailer is the most insightful thought of the day. When you're looking for backup comms, is it just going to be the ham radio operators and am/fm radio stations left if there were some outage? With tv having gone digital it's not possible to tune in and pick up the audio carrier anymore. Wartime and times of civil unrest the first thing you do is take over communication to the citizens. Without your internet^Wpodcast of the news, how will you know what is going on? If redundancy is sacrificed in the name of better quarterly earnings is it the right decision? There is a problem with this thinking, so in case of an emergency you expect to switch and change how you do things?! That will not work, as we can barely make it work under *non_emergency_conditions*. The strategy has too be that things contine to work as they used to do even in an emergency. this is not only interesting from a network operators perspective but from a governance perspective as well. I've not done any ham radio stuff for ~15+ years but do keep a shortwave radio around (battery powered of course). Ham's can do orderwire, but not replace for example a IP network, if you are lucky, you get kilobits on shoer wave with 10e-5 BER.. The first thing to happen will be the network will be severed. Look at what happened in Burma. Both their internet links were turned off, and not just taking down BGP, but the circuits were unplugged. The best netweok is the one that never works right, so you excercise the redundancy all the time.. -P
Re: Fiber cut in SF area
Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. This completely ignores the grooming problem. Not completely; it just gives you teeth for exiting your contract earlier and finding a more responsible provider to go with who won't violate the terms of the contract and re-groom you without proper notification. That's a post-facto financial recovery / liability limitation technique, not a high availability / hardening technique... I'll admit I'm somewhat simplifying the scenario, in that I also insist on no single point of failure, so even an entire site going dark doesn't completely knock out service; those who have been around since the early days will remember my email to NANOG about the gas main cut in Santa Clara that knocked a good chunk of the area's connectivity out, *not* because the fiber was damaged, but because the fire marshall insisted that all active electrical devices be powered off (including all UPSes) until the gas in the area had dissipated. Ever since then, I've just acknowledged you can't keep a single site always up and running; there *will* be events that require it to be powered down, and part of my planning process accounts for that, as much as possible, via BCP planning. I was less than a mile away from that, I remember it well. My corner cube even faced in that direction. I heard the noise then the net went poof. One of those Oh, that's not good at all combinations. Now, I'll be the first to admit it's a different game if you're providing last-mile access to single-homed customers. But sitting on the content provider side of the fence, it's entirely possible to build your infrastructure such that having 3 or more OC192s cut at random places has no impact on your ability to carry traffic and continue functioning. You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business. You simply count on them not playing entirely fair, and penalize them when they don't; and you have enough parallel contracts with different providers at different sites that outages don't take you completely offline. The problem with grooming is that in many cases, due to provider consolidation and fiber vendor consolidation and cable swap and so forth, you end up with parallel contracts with different providers at different sites that all end up going through one fiber link anyways. I had (at another site) separate vendors with fiber going northbound and southbound out of the two diverse sites. Both directions from both sites got groomed without notification. Slightly later, the northbound fiber was Then rerouted a bit up the road, into a southbound bundle (same one as our now-groomed southbound link), south to another datacenter then north again via another path. To improve route reduncancy northbound overall, for the providers' overall customer links. And the shared link south of us was what got backhoed. This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. -george william herbert gherb...@retro.com
Re: Fiber cut in SF area
On Tue, Apr 14, 2009 at 03:41:25AM +0200, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P Yo, Peter. You speak of infrastructure as if it was a monolithic thing. Why would you think that some localized NoCal fiber cuts would be taking out the whole countrys infrastructure? --bill
Re: Fiber cut in SF area
On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: [much material snipped in the interests of saving precious electron resources...] This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. Ultimately, I think a .sig line I saw years back summed it up very succinctly: Earth is a single point of failure. Below that, you're right, we're all just quibbling about which digits to put to the right of the decimal point. If the entire west coast of the US drops into the ocean, yes, having my data backed up on different continents will help; but I'll be swimming with the sharks at that point, and won't really be able to care much, so the extent of my disaster planning tends to peter out around the point where entire states disappear, and most definitely doesn't even wander into the realm of entire continents getting cut off, or the planet getting incinerated in a massive solar flare. Fundamentally, though, I think it's actually good we have outages periodically; they help keep us employed. When networks run too smoothly, management tends to look upon us as unnecessary overhead that can be trimmed back during the next round of layoffs. The more they realize we're the only bulwark against the impending forces of chaos you mentioned above, the less likely they are to trim us off the payroll. Matt Note--tongue was firmly planted in cheek; no slight was intended against those who may have lost jobs recently; post was intended for humourous consumption only; any resemblence to useful content was purely coincidental and not condoned by any present or past employer. Repeated exposure may be habit forming. Do not read while operating heavy machinery.
Re: Fiber cut in SF area
Rofl Matt, I was recently laid off from my job for 'economic' reasons, what you say is deadly accurate. Bravo! :) On Mon, Apr 13, 2009 at 7:01 PM, Matthew Petach mpet...@netflight.comwrote: On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: [much material snipped in the interests of saving precious electron resources...] This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. Ultimately, I think a .sig line I saw years back summed it up very succinctly: Earth is a single point of failure. Below that, you're right, we're all just quibbling about which digits to put to the right of the decimal point. If the entire west coast of the US drops into the ocean, yes, having my data backed up on different continents will help; but I'll be swimming with the sharks at that point, and won't really be able to care much, so the extent of my disaster planning tends to peter out around the point where entire states disappear, and most definitely doesn't even wander into the realm of entire continents getting cut off, or the planet getting incinerated in a massive solar flare. Fundamentally, though, I think it's actually good we have outages periodically; they help keep us employed. When networks run too smoothly, management tends to look upon us as unnecessary overhead that can be trimmed back during the next round of layoffs. The more they realize we're the only bulwark against the impending forces of chaos you mentioned above, the less likely they are to trim us off the payroll. Matt Note--tongue was firmly planted in cheek; no slight was intended against those who may have lost jobs recently; post was intended for humourous consumption only; any resemblence to useful content was purely coincidental and not condoned by any present or past employer. Repeated exposure may be habit forming. Do not read while operating heavy machinery. -- Respectfully, Chris Hart George Carlinhttp://www.brainyquote.com/quotes/authors/g/george_carlin.html - Frisbeetarianism is the belief that when you die, your soul goes up on the roof and gets stu...
Re: Fiber cut in SF area
On Tue, Apr 14, 2009 at 03:41:25AM +0200, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P Yo, Peter. You speak of infrastructure as if it was a monolithic thing. Why would you think that some localized NoCal fiber cuts would be taking out the whole countrys infrastructure? --bill If you are talking residential access, in the future when people work from home, the study we did in 2000 came down to that you can only loose 30 subs on a single-point-of failure tehing, and the recomendation was to interlave them, so your neighbour would have connectivity. While on this, we have an even bigger problem, the impact of loosing power is bigger, but their system has not gained the same amount of complexity as ours in the last 100 years.. (the book from 1907 on power-lines is still applicable.) -P
Re: Fiber cut in SF area
telmn...@757.org wrote: Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Better they cut the fiber instead of Oklahoma Citying the central office. If you're referring to the Event, that scares me every day about the largest meet points in the nation and how much traffic can really fully switch to other paths should one or two disappear completely. On the data side of things, though, while it still takes time, I'm forever impressed at how fast everything comes together to get communications rolling again. Man-made or natural, disasters bring out the best and the worst. Of course, I mostly see natural disasters; wasn't far from the tornado that decorated the Tandy building in Ft. Worth, was 5 miles from the Tornado in Moore, OK, and was bunkered down in my house in Lone Grove this year. I've seen 2 man-made disasters and 2 natural disasters so far this year. One was severe at a network level (Building power outage because the NOC chose not to check it out and discover the faulty power transfer switch; batteries died 8 hours later), and 3 were local and only effected a subset of end users due to cable damage (Tornado in Lone Grove back in Feb, wildfires last week in Lone Grove, and one of our nearby towns had an oversized truck grab the overhead cable and drag it down the road, ripping poles out of the ground, and of course he didn't stick around to pay the bill). If you're referring to our infrastructure, no comment but lots of laughter. I really haven't considered the SF fiber cut to be a big deal. It may effect more people, but it's still a couple minor cuts. From the back woods, Jack
Re: Fiber cut in SF area
On Apr 13, 2009, at 8:40 PM, telmn...@757.org wrote: Better they cut the fiber instead of Oklahoma Citying the central office. I'm not sure that the someone will alway s find the weakest link argument can be summed up any better than this. If you don't believe it, you all need to spend more time in the big room with the blue ceiling outside of your colos/DCs. Daryl
Re: Fiber cut in SF area
On Sat, 11 Apr 2009, Joe Greco wrote: Public key crypto is, pretty much by definition, reliant on the obscurity of private keys in order to make it work. In security terms, public key crypto is not security by obscurity, as the obscurity part is related to how the method works, and the key is secret. So openssh is definitely not security by obscurity, as anyone with programming knowledge can find out exactly how everything works, and the only thing that is a secret is the private key generated. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Fiber cut in SF area
On Sat, 11 Apr 2009, Christopher Morrow wrote: I'm not sure that the manholes == atm discussion is valid, but in the end the same thing is prone to happen to the manholes, there isn't going to be a unique key per manhole, at best it'll be 1/region or 1/manhole-owner. In the end that key is compromised as soon as the decision is made :( Also keep in mind that keyed locks don't really provide much protection, since anyone can order lockpicks over the interwebs these days, even to states where ownership is apparently illegal :( Too bad there isn't 1Password for manhole covers. --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Fiber cut in SF area
Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Thanks, *excellent* example. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Of course, but I said if we accept that. It was a challenge for the previous poster. ;-) Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. That's really it, and bringing us back to the fiber discussion, we are forced, generally, to rely on obscurity. In general, talk to a hundred people on the street, few of them are likely to be able to tell you how fiber gets from one city to another, or that a single fiber may be carrying immense amounts of traffic. Most people expect that it just all works somehow. The fact that it's buried means that it is sufficiently inaccessible to most people. It will still be vulnerable to certain risks, including backhoes, anything else that disrupts the ground (freight derailments, earthquakes, etc), but those are all more or less natural hazards that you protect against with redundancy. The guy who has technical specifics about your fiber network, and who picks your vulnerable points and hits you with a hacksaw, that's just always going to be much more complex to defend against. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Fiber cut in SF area
Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Just my 2¢
Re: Fiber cut in SF area
Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least itâs a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Cute, but a lot of people seem to be wondering this, so a better answer is deserved. The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk There are all sorts of strategies for attacking ATM's, and being susceptible to a sledgehammer, crowbar, or truck smashing into the unit shouldn't be hard to understand. Most data centers have security that is designed to keep honest people out of places that they shouldn't be. Think that security guard at the front will stop someone from running off with something valuable? Maybe. Have you considered following the emergency fire exits instead? Running out the loading dock? Etc? Physical security is extremely difficult, and defending against a determined, knowledgeable, and appropriately resourced attacker out to get *you* is a losing battle, every time. Think about a door. You can close your bathroom door and set the privacy lock, but any adult with a solid shoulder can break that door, or with a pin (or flathead or whatever your particular knob uses) can stick it in and trigger the unlock. Your front door is more solid, but if it's wood, and not reinforced, I'll give my steel-toed boots better than even odds against it. What? You have a commercial hollow steel door? Ok, that beats all of that, let me go get my big crowbar, a little bending will let me win. Something more solid? Ram it with a truck. You got a freakin' bank vault door? Explosives, torches, etc. Fort Knox? Bring a large enough army, you'll still get in. Notice a pattern? For any given level of protection, countermeasures are available. Your house is best secured by making changes that make it appear ordinary and non-attractive. That means that a burglar is going to look at your house, say nah, and move on to your neighbor's house, where your neighbor left the garage open. But if I were a burglar and I really wanted in your house? There's not that much you could really do to stop me. It's just a matter of how well prepared I am, how well I plan. So. Now. Fiber. Here's the thing, now. First off, there usually isn't a financial motivation to attack fiber optic infrastructure. ATM's get some protection because without locks, criminals would just open them and take the cash. Having locks doesn't stop that, it just makes it harder. However, the financial incentive for attacking a fiber line is low. Glass is cheap. We see attacks against copper because copper is valuable, and yet we cannot realistically guard the zillions of miles of copper that is all around. Next. Repair crews need to be able to access the manholes. This is a multifaceted problem. First off, since there are so many manholes to protect, and there are so many crews who might potentially need to access them, you're probably stuck with a standardized key approach if you want to lock them. While this offers some protection against the average person gaining unauthorized access, it does nothing to prevent inside job attacks (and I'll note that this looks suspiciously like an inside job of some sort). Further, any locking mechanism can make it more difficult to gain access when you really need access; some manholes are not opened for years or even decades at a time. What happens when the locks are rusted shut? Is the mechanism weak enough that it can be forced open, or is it tolerable to have to wait extra hours while a crew finds a way to open it? Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. Are you going to protect the concrete too? If not, what prevents me from simply breaking away the concrete around the manhole cover rim (admittedly a lot of work) and just discarding the whole thing? Wait. I just want to *break* the cable? Screw all that. Get me a backhoe. I'll just eyeball the direction I think the cable's going, and start digging until I snag something. Start to see the problems? I'm not saying that security is a bad thing, just a tricky thing. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing
Re: Fiber cut in SF area
Once upon a time, Jo¢ jbfixu...@gmail.com said: Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno We've had several occasions here where somebody has stolen a backhoe or front-end loader from a construction site, driven to the nearest ATM, and loaded the whole ATM into a (usually stolen) truck. Also, what is the density of outdoor ATMs? I'm in a suburban area, and there may be one every mile or two. How large is the fiber plant? Miles and miles of continuous fiber, every inch of which is equally important. A lot of it here is even on poles, not buried. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: Fiber cut in SF area
* Joe Greco: The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk Heh. Once you install ATMs into solid walls, the attacks get a tad more interesting. In some places of the world, gas detectors are almost mandatory because criminals pump gas into the machine, ignite it, and hope that the explosion blows a hole into the machine without damaging the money (which seems to work fairly well if you use the right gas at the right concentration).
Re: Fiber cut in SF area
On Sat, Apr 11, 2009 at 11:10 AM, Florian Weimer f...@deneb.enyo.de wrote: * Joe Greco: The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk Heh. Once you install ATMs into solid walls, the attacks get a tad more interesting. In some places of the world, gas detectors are almost mandatory because criminals pump gas into the machine, ignite it, and hope that the explosion blows a hole into the machine without damaging the money (which seems to work fairly well if you use the right gas at the right concentration). also, there is the fact that some very large percentage of ATM machines were installed with the same admin passwd setup. I recall ~1.5 yrs ago some news about this, and that essentially banks send out the ATM machines with a stock passwd (sometimes the default which is documented in easily google-able documents) per bank (BoFA uses passwd123, Citi uses passwd456 ) I'm not sure that the manholes == atm discussion is valid, but in the end the same thing is prone to happen to the manholes, there isn't going to be a unique key per manhole, at best it'll be 1/region or 1/manhole-owner. In the end that key is compromised as soon as the decision is made :( Also keep in mind that keyed locks don't really provide much protection, since anyone can order lockpicks over the interwebs these days, even to states where ownership is apparently illegal :( -Chris
Re: Fiber cut in SF area
The best protecion is good engineering taking advantage of technologies and architecures available since long time ago at any of the different network layers. Why network operators/carriers don't do it ?, it's another issue and most of the time is a question of bottom line numbers for which there are no engineering solutions. My .02
[OT] Re: Fiber cut in SF area
On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. You can cut the concrete, too, for that matter, with oxyacetylene, as long as you wear certain protective gear. We have a few vault covers here that are concrete covering the largest vaults we have. You need more than a manhole hook to get one of those covers up. The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. One way of making a site more resistant to 'inside job' issues is with SCIF- like controls (see http://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility ) and using combination locks such as the Sargent and Greenleaf 8077AD for control, and the SG 833 superpadlock for security (see http://www.sargentandgreenleaf.com/PL-833.php ). The tech would have the 833's key, and the area supervisor the combination. The 8077AD's combination is very easily changed in the field, and could be changed frequently. The key to this method's success is that the keyholder to the 833 cannot have the combination, and the holder of the combination cannot have an 833 key. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost
Re: [OT] Re: Fiber cut in SF area
You can cut the concrete, too, for that matter, with oxyacetylene, as long as you wear certain protective gear. We have a few vault covers here that are concrete covering the largest vaults we have. You need more than a manhole hook to get one of those covers up. And when you think you have it safely burried someone drives a tunnel boring machine through it - http://www.flickr.com/photos/23919...@n00/3426407496/ brandon
RE: Fiber cut in SF area
Jo? wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it?s a start. That is an option, but it doesn't address the real problem. The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. No one should be surprised that ATT would cut-corners on critical infrastructure. The good news is that this incident will likely result in increased Federal scrutiny if not regulation. We know how spectacularly energy and banking deregulation failed. Is that mistake being repeated with telecommunications? The bad news is that some of the $16M/yr ATT spends lobbying Congress (for things like fighting number portability and getting a free pass on illegal domestic surveillance) will likely be redirected to ask for money to fix the problem they created. This assumes ATT is as badly managed, and the US FCC and DHS are better managed, than has been the case for the last 8 years. Time will tell. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: http://www.jivamedia.com/askdrdawn/askdrdawn.php http://geekspeak.org/ Roger Marquis
Re: Fiber cut in SF area
The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. BBN won the contract to build the first four IMPs. Theory and research about it is older, look at: http://www.lk.cs.ucla.edu/LK/Bib/REPORT/PhD/proposal-01.html But you are right, redundancy is the issue, cost is the factor. Jorge.
Re: [OT] Re: Fiber cut in SF area
On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. Yes, but we were discussing locked covers, which (given the underlying assumptions of this discussion) might be a bit heavier. Further, it would be vaguely suspicious and more noticeable for a road crew or power company truck to be deploying such gear, might draw more attention. The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. More expense. :-) One way of making a site more resistant to 'inside job' issues is with SCIF- like controls (see http://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility ) and using combination locks such as the Sargent and Greenleaf 8077AD for control, and the SG 833 superpadlock for security (see http://www.sargentandgreenleaf.com/PL-833.php ). The tech would have the 833's key, and the area supervisor the combination. The 8077AD's combination is very easily changed in the field, and could be changed frequently. The key to this method's success is that the keyholder to the 833 cannot have the combination, and the holder of the combination cannot have an 833 key. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost Certainly it is *possible* to do it, but given the other variables, does it make *sense*? Consider what I was saying about just going to town with a backhoe. You have a lot to protect. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
RE: Fiber cut in SF area
On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: Why didn't the man in the street pharmacy have its own backup plans? Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides ATT or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions? Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy. Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc. Readiness works better if everyone does their part, including grandma. Next time it won't be ATT, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or or or . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, Everything fails sometimes. What's your plan? http://www.ready.gov/ personal opinion only
Re: Fiber cut in SF area
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate. The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer. Sorry for the random thoughts... -Mike On Sat, Apr 11, 2009 at 4:11 PM, Sean Donelan s...@donelan.com wrote: On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: Why didn't the man in the street pharmacy have its own backup plans? Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides ATT or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions? Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy. Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc. Readiness works better if everyone does their part, including grandma. Next time it won't be ATT, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or or or . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, Everything fails sometimes. What's your plan? http://www.ready.gov/ personal opinion only
Re: Fiber cut in SF area
While OT the news reports indicated ATMs were offline and many credit card processing machines were down. This is no big shock because many ATM networks are on frame relay and POS credit card machines use POTS lines. The outage also impacted mobile service too if it hadn't been said. I hope we can put this thread to rest soon. -r On Sat, Apr 11, 2009 at 04:25:26PM -0700, Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate. The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer. Sorry for the random thoughts... -Mike
Re: Fiber cut in SF area
Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. ... Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Re: Fiber cut in SF area
Don't really care so much about the bank's security, especially if it was one that received some the bailout money :) I was more worried about if people could make withdraws from their bank accounts. Deposits they could do as they could enter them in later but withdraws I think would be different. On Sat, Apr 11, 2009 at 5:19 PM, Roy r.engehau...@gmail.com wrote: Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. ... Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Re: Fiber cut in SF area
Sean Donelan wrote: Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.
Re: [OT] Re: Fiber cut in SF area
On Sat, Apr 11, 2009 at 2:43 PM, Joe Greco jgr...@ns.sol.net wrote: On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. Yes, but we were discussing locked covers, which (given the underlying assumptions of this discussion) might be a bit heavier. Further, it would be vaguely suspicious and more noticeable for a road crew or power company truck to be deploying such gear, might draw more attention. Cop: 'What are you fellows doing there with the torch? Me: Us? Oh yea some dipstick plugged up our lock here with epoxy, our quick solution cause of the outage is to cut the lock/blah off with a torch, bummer, eh? I hate dipsticks... Cop: Cool, have a good night! :( The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. More expense. :-) and complexity and parts to lose and people to have away during normal outage repairs and ... :( fail. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost Certainly it is *possible* to do it, but given the other variables, does it make *sense*? Consider what I was saying about just going to town with a backhoe. You have a lot to protect. and I also would ask.. what's the cost/risk here? 'We' lost at best ~1day for some folks in the outage, nothing global and nothing earth-shattering... This has happened (this sort of thing) 1 time in how many years? Expending $$ and time and people to go 'put padlocks on manhole covers' seems like spending in the wrong place... (yes, I agree also that simply dropping into a manhole with an axe/hacksaw is pretty simple to do, it's also just about impossible to realisitcally protect against) -Chris
RE: Fiber cut in SF area
I know as far as att/sbc/pacbell a lot of the time they run the ring within the same conduit to at least have hardware protection on the circuit I'm sure it's the same with other providers. -carlos -Original Message- From: Roy [mailto:r.engehau...@gmail.com] Sent: Saturday, April 11, 2009 6:02 PM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Sean Donelan wrote: Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.
Re: Fiber cut in SF area
Jorge Amodio wrote: s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. /DARPA/ARPA/ may be splitting hairs. According to http://www.livinginternet.com/i/ii_roberts.htm DARPA head Charlie Hertzfeld promised IPTO Director Bob Taylor a million dollars to build a distributed communications network. And apologies WRT /BBM/BBN/. Guess it was really has been a while now (given the 4 and 5 figure checks to BBN I signed back in the day). Sean Donelan wrote: On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. Apparently even some network operators don't yet grasp the significance of this event. Why didn't the man in the street pharmacy have its own backup plans? I assume they, as most of us, believed the government was taking care of the country's critical infrastructure. Interesting how well this illustrates the growing importance of the Internet vis-a-vis other communications channels. Roger Marquis
Re: [OT] Re: Fiber cut in SF area
Christopher Morrow morrowc.li...@gmail.com writes: and I also would ask.. what's the cost/risk here? 'We' lost at best ~1day for some folks in the outage, nothing global and nothing earth-shattering... This has happened (this sort of thing) 1 time in how many years? Expending $$ and time and people to go 'put padlocks on manhole covers' seems like spending in the wrong place... as long as the west's ideological opponents want terror rather than panic, and also to inflict long term losses rather than short term losses, that's true. in this light you can hopefully understand why bollards to protect internet exchanges against truck bombs are not only penny wise pound foolish (since the manholes a half mile away won't be hardened or monitored or even locked) but also completely wrongheaded (since terrorists need publicity which means they need their victims to be fully able to communicate.) -- Paul Vixie
Re: Fiber cut in SF area
An easy way to describe what your saying is Security by obscurity is not security On Apr 11, 2009, at 8:31 AM, Joe Greco wrote: Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Cute, but a lot of people seem to be wondering this, so a better answer is deserved. The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk There are all sorts of strategies for attacking ATM's, and being susceptible to a sledgehammer, crowbar, or truck smashing into the unit shouldn't be hard to understand. Most data centers have security that is designed to keep honest people out of places that they shouldn't be. Think that security guard at the front will stop someone from running off with something valuable? Maybe. Have you considered following the emergency fire exits instead? Running out the loading dock? Etc? Physical security is extremely difficult, and defending against a determined, knowledgeable, and appropriately resourced attacker out to get *you* is a losing battle, every time. Think about a door. You can close your bathroom door and set the privacy lock, but any adult with a solid shoulder can break that door, or with a pin (or flathead or whatever your particular knob uses) can stick it in and trigger the unlock. Your front door is more solid, but if it's wood, and not reinforced, I'll give my steel-toed boots better than even odds against it. What? You have a commercial hollow steel door? Ok, that beats all of that, let me go get my big crowbar, a little bending will let me win. Something more solid? Ram it with a truck. You got a freakin' bank vault door? Explosives, torches, etc. Fort Knox? Bring a large enough army, you'll still get in. Notice a pattern? For any given level of protection, countermeasures are available. Your house is best secured by making changes that make it appear ordinary and non-attractive. That means that a burglar is going to look at your house, say nah, and move on to your neighbor's house, where your neighbor left the garage open. But if I were a burglar and I really wanted in your house? There's not that much you could really do to stop me. It's just a matter of how well prepared I am, how well I plan. So. Now. Fiber. Here's the thing, now. First off, there usually isn't a financial motivation to attack fiber optic infrastructure. ATM's get some protection because without locks, criminals would just open them and take the cash. Having locks doesn't stop that, it just makes it harder. However, the financial incentive for attacking a fiber line is low. Glass is cheap. We see attacks against copper because copper is valuable, and yet we cannot realistically guard the zillions of miles of copper that is all around. Next. Repair crews need to be able to access the manholes. This is a multifaceted problem. First off, since there are so many manholes to protect, and there are so many crews who might potentially need to access them, you're probably stuck with a standardized key approach if you want to lock them. While this offers some protection against the average person gaining unauthorized access, it does nothing to prevent inside job attacks (and I'll note that this looks suspiciously like an inside job of some sort). Further, any locking mechanism can make it more difficult to gain access when you really need access; some manholes are not opened for years or even decades at a time. What happens when the locks are rusted shut? Is the mechanism weak enough that it can be forced open, or is it tolerable to have to wait extra hours while a crew finds a way to open it? Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. Are you going to protect the concrete too? If not, what prevents me from simply breaking away the concrete around the manhole cover rim (admittedly a lot of work) and just discarding the whole thing? Wait. I just want to *break* the cable? Screw all that. Get me a backhoe. I'll just eyeball the direction I think the cable's going, and start digging until I snag something. Start to see the problems? I'm not saying that security is a bad thing, just a tricky thing. ... JG -- Joe Greco - sol.net Network Services
Re: Fiber cut in SF area
Roger Marquis wrote: Why didn't the man in the street pharmacy have its own backup plans? I assume they, as most of us, believed the government was taking care of the country's critical infrastructure. Interesting how well this illustrates the growing importance of the Internet vis-a-vis other communications channels. It's also possible that they just planned on being down in such an event. There's two factors here: Not all low frequency risks are worth mitigating (how many of us have generators at home). Humans are bad at planning around rare events. Econimist Nassim Taleb's book The Black Swan (isbn 978-1400063512) ought to be on everyones list for coverage of the subject matter. Fiber cuts are well outside the realm of experience for your average business manager. The normal remediation strategy (for telecommunications outage) in fact worked just fine, call your provider, and or wait for them to fix it. Roger Marquis
Re: [OT] Re: Fiber cut in SF area
On Sat, 11 Apr 2009, Lamar Owen wrote: The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. I'm starting to wonder what makes more sense -- locking down thousands of miles of underground tunnel with mil-spec expensive locks that ideally keep unauthorized people out, OR simple motion and or video cameras in the tunnels themselves which relay their access back to a central facility, along with a video feed of sorts, to help identify who is there, whether approved or not. With locks, you know they gained access after the fact and that your locking wasn't sufficient enough. With active monitoring of the area where the cables live, you at least know the moment someone goes in, and have some lead time (and maybe a video) to do something to prevent it, or catch them in the act. Unfortunately, that kind of monitoring is also expensive and complex. I wonder what the cost of the outage was, and how much it might cost to monitor it? Would it be worth $2,000 per site per year? A great webcam, with day/night capability, and a cell phone, in a locked box, with a solar panel, on top of a pole, near the site. Sure, if you know it's there, taking it out is easy, but someone will still know something is wrong when it goes dark or the picture changes significantly. Are there some low-cost, highly-effective ways that the tunnels which carry our precious data and communications can at least be monitored remotely? Waiting for someone to cut a cable and then deploying a crew seems reactive, whereas knowing the moment someone goes INTO the tunnel is proactive, whether the person(s) are there to do some normal maintenance or something malicious. Beckman I suppose rats and other rodents could cause such a system to be too annoying to pay attention to. --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Fiber cut in SF area
An easy way to describe what your saying is Security by obscurity is not security Yes and no. From a certain point of view, security is almost always closely tied to obscurity. A cylinder lock is simply a device that operates through principles that are relatively unknown to the average person: they just know that you stick a key in, turn it, and it opens. The security of such a lock is dependent on an attacker not knowing what a pin and tumbler design is, and not having the tools and (trivial) skills needed to defeat it. That is obscurity of one sort. Public key crypto is, pretty much by definition, reliant on the obscurity of private keys in order to make it work. Ouch, eh. And hard to obtain is essentially a parallel as well. Simply making keyblanks hard to obtain is really a form of obscurity. How much security is dependent on that sort of strategy? It can (and does) work well in many cases, but knowing the risks and limits is important. But that's all assuming that you're trying to secure something against a typical attacker. My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Fiber cut in SF area
Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. Mike
Re: Fiber cut in SF area
George William Herbert wrote: Scott Doty wrote: (Personally, I can think of a MAE-Clueless episode that was worse than this, but that was in the 90's...) The gas main strike out front of the building in Santa Clara? Or something else? -george william herbert gherb...@retro.com Hi George, No, it was when an AS took their full bgp feed fed it into their igp (which used RIP, iirc), which generated (de-aggregated) routes into /24's, which they then announced back into bgp... iirc, part of the chaos than ensued was due to a router bug, so that the routes stuck around in global views, even after the AS killed their announcements, and even after physically disconnecting from their provider. We told our customers the Internet is broken, please try again later...which was acceptable back then. (But I doubt we would get away with just that nowadays... ;-) ) -Scott
Re: Fiber cut in SF area
On Apr 10, 2009, at 3:41 PM, Scott Doty wrote: George William Herbert wrote: Scott Doty wrote: (Personally, I can think of a MAE-Clueless episode that was worse than this, but that was in the 90's...) The gas main strike out front of the building in Santa Clara? Or something else? -george william herbert gherb...@retro.com No, it was when an AS took their full bgp feed fed it into their igp (which used RIP, iirc), which generated (de-aggregated) routes into /24's, which they then announced back into bgp... That was Vinny Bono of FLIX, the Fat man Little man Internet eXchange, as7007. Happened in 1997, IIRC. He used a Bay Networks router to redistribute BGP on one card into RIPv1 on another card, stripping the CIDR notations off each prefix, making them classful, and stripping the AS Path. This means, for instance, 96.0.0.0 was a /8, not a /24. It also means He then re-redistributed RIP into BGP on a third card, which then originated each route from as7007. I have it on most excellent authority (the Fat man himself) that this was not possible on ciscos. Wonder if it is now ... ? Anyway, I did not know people were calling this the MAE-Clueless incident. I've always called it the 7007 incident. In fact, some people still have as7007 filtered. iirc, part of the chaos than ensued was due to a router bug, so that the routes stuck around in global views, even after the AS killed their announcements, and even after physically disconnecting from their provider. That was Sprint, as7007's transit provider. Sprint only did AS Path filtering, and as every single prefix was ^7007$, they all passed the filter. Vinny literally unplugged the router, no power, no fiber, no copper, but the prefixes were still bouncing around the 'Net for hours. Sprint kept the routes around for a long time as their routers would not honor withdrawals - or so the rumors said. The rumors also claimed the IOS version was named $FOO-sean. Sean Doran was CTO of Sprint's Internet company at the time, and he supposedly specifically asked for the 'feature' of ignoring withdrawals to lower CPU on their AGS+s. I have absolutely no way of confirming this as I haven't spoken to Sean in years years, and wouldn't even know where to find him any more. The most interesting rumor I heard is that Sprint had to shut down every single router simultaneously to clear the routes out of their network. Personally I think that's probably a bit exaggerated, but who knows? We told our customers the Internet is broken, please try again later...which was acceptable back then. (But I doubt we would get away with just that nowadays... ;-) ) Really? That's what some broadband providers say nearly daily. -- TTFN, patrick
RE: Fiber cut in SF area
I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least its a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno Just my 2¢
Re: Fiber cut in SF area
Yup. Abovenet fiber between 200 Paul SFO and 11 Great Oaks SJC is currently out of commission. jason On Thu, Apr 9, 2009 at 11:37 AM, Stefan Molnar ste...@csudsu.com wrote: VZ in the South Bay (San Jose) is out. As per news reports I watched at 6am PDT. --Original Message-- From: Craig Holland To: NANOG Subject: Fiber cut in SF area Sent: Apr 9, 2009 8:14 AM Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig
Re: Fiber cut in SF area
200 Paul Ave is seeing several carriers down. I am also in Santa Cruz and cannot make or receive long distance calls on my land lines. Unconfirmed reports of Caltrain cut. Cheers, Aaron On Thu, Apr 09, 2009 at 03:37:14PM +, Stefan Molnar wrote: VZ in the South Bay (San Jose) is out. As per news reports I watched at 6am PDT. --Original Message-- From: Craig Holland To: NANOG Subject: Fiber cut in SF area Sent: Apr 9, 2009 8:14 AM Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig -- Aaron Hughes aar...@bind.com (703) 244-0427 Key fingerprint = AD 67 37 60 7D 73 C5 B7 33 18 3F 36 C3 1C C6 B8 http://www.bind.com/
Re: Fiber cut in SF area
Hello, Mercurynews.com is reporting telephone outages in Santa Clara and Santa Cruz counties that started around 2:00 am local time. I observed numerous carrier outages starting around 4:00 am local time. Does anyone know if this is due to the same fiber cut, or are these separate issues? David At 10:12 AM 4/9/2009, you wrote: 200 Paul Ave is seeing several carriers down. I am also in Santa Cruz and cannot make or receive long distance calls on my land lines. Unconfirmed reports of Caltrain cut. Cheers, Aaron On Thu, Apr 09, 2009 at 03:37:14PM +, Stefan Molnar wrote: VZ in the South Bay (San Jose) is out. As per news reports I watched at 6am PDT. --Original Message-- From: Craig Holland To: NANOG Subject: Fiber cut in SF area Sent: Apr 9, 2009 8:14 AM Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig -- Aaron Hughes aar...@bind.com (703) 244-0427 Key fingerprint = AD 67 37 60 7D 73 C5 B7 33 18 3F 36 C3 1C C6 B8 http://www.bind.com/
RE: Fiber cut in SF area
Seeing the same thing have an oc48 down from abovenet out of 200 paul -carlos -Original Message- From: Aaron Hughes [mailto:aar...@bind.com] Sent: Thursday, April 09, 2009 9:13 AM To: Stefan Molnar Cc: NANOG Subject: Re: Fiber cut in SF area 200 Paul Ave is seeing several carriers down. I am also in Santa Cruz and cannot make or receive long distance calls on my land lines. Unconfirmed reports of Caltrain cut. Cheers, Aaron On Thu, Apr 09, 2009 at 03:37:14PM +, Stefan Molnar wrote: VZ in the South Bay (San Jose) is out. As per news reports I watched at 6am PDT. --Original Message-- From: Craig Holland To: NANOG Subject: Fiber cut in SF area Sent: Apr 9, 2009 8:14 AM Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig -- Aaron Hughes aar...@bind.com (703) 244-0427 Key fingerprint = AD 67 37 60 7D 73 C5 B7 33 18 3F 36 C3 1C C6 B8 http://www.bind.com/
Re: Fiber cut in SF area
On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote: Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Confirmed VZW ATT; http://cbs5.com/local/phone.internet.outage.2.980578.html Rather widespread general telco outage, the county has deployed extra patrol units in the south bay to compensate for not being able to call 911. Third video link in shows repairs underway. -- David W. HankinsIf you don't do it right the first time, Software Engineeryou'll just have to do it again. Internet Systems Consortium, Inc. -- Jack T. Hankins pgp3AV5KN6ukx.pgp Description: PGP signature
Re: Fiber cut in SF area
News coverage: http://cow.org/r/?5459 http://cow.org/r/?545a And not that I expect any useful updates: http://twitter.com/attnews -r On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote: Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig
Re: Fiber cut in SF area
Hi, On Thu, Apr 09, 2009 at 11:15:05AM -0600, David Edwards wrote: Mercurynews.com is reporting telephone outages in Santa Clara and Santa Cruz counties that started around 2:00 am local time. I observed numerous carrier outages starting around 4:00 am local time. Does anyone know if this is due to the same fiber cut, or are these separate issues? This seems to be due to the same fiber cut when following local news and scanner frequencies. -andreas -- Andreas Ott K6OTT andr...@naund.org
Re: Fiber cut in SF area
I saw my Sonic.net-over-ATT ADSL go dark at 02:30 local and it is still down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul outages are associated with where this ATM terminates and that's the cause, rather than the service in/out of Santa Cruz County, but I have no way of telling which from here) My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise there), and I'm not seeing significant routing problems in/out of there with transit or peering. (Not even any down peers, so no inter-Equinix-site outage apparently). Matthew Kaufman matt...@eeph.com
RE: Fiber cut in SF area
Level3 is having problems in the 216 area code as well (Cleveland) George Roettger -Original Message- From: David Edwards [mailto:da...@reliablehosting.com] Sent: Thursday, April 09, 2009 1:15 PM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Hello, Mercurynews.com is reporting telephone outages in Santa Clara and Santa Cruz counties that started around 2:00 am local time. I observed numerous carrier outages starting around 4:00 am local time. Does anyone know if this is due to the same fiber cut, or are these separate issues? David
Re: Fiber cut in SF area
Anyone know where the actual cut is? On 4/9/09, David W. Hankins david_hank...@isc.org wrote: On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote: Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Confirmed VZW ATT; http://cbs5.com/local/phone.internet.outage.2.980578.html Rather widespread general telco outage, the county has deployed extra patrol units in the south bay to compensate for not being able to call 911. Third video link in shows repairs underway. -- David W. Hankins If you don't do it right the first time, Software Engineer you'll just have to do it again. Internet Systems Consortium, Inc. -- Jack T. Hankins -- Sent from my mobile device
Re: Fiber cut in SF area
isn't there a mailing list for this sort of thing? outages@ I think it is? (not that I mind, just a little advert for the appropriate forum, and a place that MAY have some useful info on this topic) -chris On Thu, Apr 9, 2009 at 1:51 PM, Ravi Pina r...@cow.org wrote: News coverage: http://cow.org/r/?5459 http://cow.org/r/?545a And not that I expect any useful updates: http://twitter.com/attnews -r On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote: Just dropping a note that there is a fiber cut in the SF area (I have a metro line down). AboveNet is reporting issues and I've heard unconfirmed reports that ATT and VZW are affected as well. Rgs, craig
