RE: Outside plant protection, fiber cuts, interwebz down oh noes!
You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I agree 100 percent The clipboard makes it official... --Original Message-- From: Jamie Bowden To: Andy Ringsmuth Cc: nanog@nanog.org Subject: RE: Outside plant protection, fiber cuts, interwebz down oh noes! Sent: Apr 13, 2009 9:07 AM You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy Sent on the Now Network� from my Sprint® BlackBerry
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I know it's fun to have these sort of discussions.. however, here in Toronto anyway all of the splicers, construction people and other contractors all know each other enough to be able to spot somebody thats not auposed to be there. The city inspectors are cruising all day looking for health and safety violations, traffic inspectors are looking for issues, and thecop Maffia is making sure you have a pay duty cop. Unless you were incredibly lucky, a rogue crew at work In a chamber would be caught very quickly. On 13-Apr-09, at 9:07 AM, Jamie Bowden ja...@photon.com wrote: You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
Re: SPEEDS
Hi Thomas, Please paste me a traceroute to google.com Regards, Bruce On Monday 13 April 2009 3:45:10 pm Matikiti, Thomas wrote: Wazup Bruce - I'm a bit concerned about our speeds here even today when they are two people in the office I still find myself struggling to browse the internet due to slow speeds. We should investigate our link's performance when there is no one in the office because for it remains the same - slow. Regards, Tom The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading Received in error, send to the original sender , then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. This email is being sent out by KPMG International on behalf of the local KPMG member firm providing services to you. KPMG International is a Swiss cooperative that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International provides no services to clients. Each member firm of KPMG International is a legally distinct and separate entity and each describes itself as such. Information about the structure and jurisdiction of your local KPMG member firm can be obtained from your KPMG representative. This footnote also confirms that this e-mail message has been swept by AntiVirus software.
Re: SPEEDS
I certainly agree that its only the two you Output queue: 0/40 (size/max) 30 second input rate 16000 bits/sec, 9 packets/sec 30 second output rate 9000 bits/sec, 7 packets/sec 51056 packets input, 28961683 bytes KPMG-BR#sh clock 15:48:52.249 GMT Mon Apr 13 2009 A significant difference as compared to peak hours On Monday 13 April 2009 3:45:10 pm Matikiti, Thomas wrote: Wazup Bruce - I'm a bit concerned about our speeds here even today when they are two people in the office I still find myself struggling to browse the internet due to slow speeds. We should investigate our link's performance when there is no one in the office because for it remains the same - slow. Regards, Tom The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading Received in error, send to the original sender , then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. This email is being sent out by KPMG International on behalf of the local KPMG member firm providing services to you. KPMG International is a Swiss cooperative that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International provides no services to clients. Each member firm of KPMG International is a legally distinct and separate entity and each describes itself as such. Information about the structure and jurisdiction of your local KPMG member firm can be obtained from your KPMG representative. This footnote also confirms that this e-mail message has been swept by AntiVirus software.
Re: SPEEDS
Kindly disregard my last it was sent in error.
Re: SPEEDS
My sincerest apologies guys, this really wasn't intended to end up here. Bruce On Monday 13 April 2009 4:08:06 pm robbie.ja...@regions.com wrote: please stop posting this to nanog. much appreciated. Bruce Anthony Grobler br...@yoafrica.c To om nanog@nanog.org cc 04/13/2009 09:05 AMSubject Re: SPEEDS Please respond to br...@yoafrica.co m I certainly agree that its only the two you Output queue: 0/40 (size/max) 30 second input rate 16000 bits/sec, 9 packets/sec 30 second output rate 9000 bits/sec, 7 packets/sec 51056 packets input, 28961683 bytes KPMG-BR#sh clock 15:48:52.249 GMT Mon Apr 13 2009 A significant difference as compared to peak hours On Monday 13 April 2009 3:45:10 pm Matikiti, Thomas wrote: Wazup Bruce - I'm a bit concerned about our speeds here even today when they are two people in the office I still find myself struggling to browse the internet due to slow speeds. We should investigate our link's performance when there is no one in the office because for it remains the same - slow. Regards, Tom The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you have received this communication in error, please address with the subject heading Received in error, send to the original sender , then delete the e-mail and destroy any copies of it. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter. Opinions, conclusions and other information in this e-mail and any attachments that do not relate to the official business of the firm are neither given nor endorsed by it. KPMG cannot guarantee that e-mail communications are secure or error-free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. This email is being sent out by KPMG International on behalf of the local KPMG member firm providing services to you. KPMG International is a Swiss cooperative that serves as a coordinating entity for a network of independent firms operating under the KPMG name. KPMG International provides no services to clients. Each member firm of KPMG International is a legally distinct and separate entity and each describes itself as such. Information about the structure and jurisdiction of your local KPMG member firm can be obtained from your KPMG representative. This footnote also confirms that this e-mail message has been swept by AntiVirus software.
Re: Fiber cut in SF area
Mike Lewinski wrote: Joe Greco wrote: Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. The accepted standard is that a system is secure iff you can disclose _all_ of the details of how the system works to an attacker _except_ the private key and they still cannot get in -- and that is true of most open-standard or open-source encryption/security products due to extensive peer review and iterative improvements. What security by obscurity refers to are systems so weak that their workings cannot be exposed because then the keys will not be needed, which is true of most closed-source systems. It does _not_ refer to keeping your private keys secret. Key management is considered to be an entirely different problem. If you do not keep your private keys secure, no security system will be able to help you. S -- Stephen Sprunk God does not play dice. --Albert Einstein CCIE #3723 God is an inveterate gambler, and He throws the K5SSSdice at every possible opportunity. --Stephen Hawking smime.p7s Description: S/MIME Cryptographic Signature
Re: Fiber cut in SF area
On Mon, 13 Apr 2009 09:18:04 -0500 Stephen Sprunk step...@sprunk.org wrote: Mike Lewinski wrote: Joe Greco wrote: Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. The accepted standard is that a system is secure iff you can disclose _all_ of the details of how the system works to an attacker _except_ the private key and they still cannot get in -- and that is true of most open-standard or open-source encryption/security products due to extensive peer review and iterative improvements. What security by obscurity refers to are systems so weak that their workings cannot be exposed because then the keys will not be needed, which is true of most closed-source systems. It does _not_ refer to keeping your private keys secret. Correct. Open source and open standards are (some) ways to achieve that goal. They're not the only ones, nor are they sufficient. (Consider WEP as a glaring example of a failure of a standards process.) On the other hand, I was once told by someone from NSA that they design all of their gear on the assumption that Serial #1 of any new crypto device is delivered to the Kremlin. This principle, as applied to cryptography, was set out by Kerckhoffs in 1883; see http://www.petitcolas.net/fabien/kerckhoffs/ for details. Key management is considered to be an entirely different problem. If you do not keep your private keys secure, no security system will be able to help you. Yes. One friend of mine likens insecurity to entropy: you can't destroy it, but you can move it around. For example, cryptography lets you trade the insecurity of the link for the insecurity of the key, on the assumption that you can more easily protect a few keys than many kilometers of wire/fiber/radio. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Cart and Horse
A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.
Re: Cart and Horse
On Monday 13 April 2009 11:06:55 Roy wrote: A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims. :-) Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all.
RE: Cart and Horse
Wouldn't some authentication system be more useful than trying to lock all the manholes? Picture a system maybe using RFID or some other radio system where you walk up to manhole, wave your 'wand' (like a Mobil Speedpass), you hear a couple beeps, and you're cleared to open the manhole. Without authenticating, you can still get in, but the NOCs at local utilities and telcos are notified, maybe police as well. If you can tie access to a particular person's ID, I doubt that person will misuse it. Of course, this requires power and battery backup. On the other hand, maybe it's time to put the blame on the unions. If the saboteur is found to be a union member, maybe penalize the entire union somehow, since they're acting like a terrorist group at that point. Chuck -Original Message- From: Lamar Owen [mailto:lo...@pari.edu] Sent: Monday, April 13, 2009 11:22 AM To: nanog@nanog.org Subject: Re: Cart and Horse On Monday 13 April 2009 11:06:55 Roy wrote: A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims. :-) Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all.
RE: Fiber cut in SF area
One thing that is missing here is before we can define security we need to define the threat and the obstruction the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them. So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the Threat to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, where do we go from here? It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.eb...@crlmed.com www.consultingradiologists.com -Original Message- From: Joe Greco [mailto:jgr...@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Thanks, *excellent* example. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Of course, but I said if we accept that. It was a challenge for the previous poster. ;-) Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most
Re: Cart and Horse
This bears investigating. I live 3 blocks away. Looks like I'm going on a stroll after work tonight. Bobby Glover Director of Information Services South Valley Interet (AS4307) - Original Message - From: Roy r.engehau...@gmail.com To: nanog na...@merit.edu Sent: Monday, April 13, 2009 8:06 AM Subject: Cart and Horse A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.
Re: Cart and Horse
Yes, they could create a solution for this that will cost money, or they could just take out the welding specs and go to town for a fraction of the price. This type of stuff is typical of incident response... Fix the bleeding and create a long term solution that won't be as big of an impact. Regards, James Pleger e: jple...@gmail.com g: http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x9D7141C9 On Apr 13, 2009, at 8:49 AM, Church, Charles wrote: Wouldn't some authentication system be more useful than trying to lock all the manholes? Picture a system maybe using RFID or some other radio system where you walk up to manhole, wave your 'wand' (like a Mobil Speedpass), you hear a couple beeps, and you're cleared to open the manhole. Without authenticating, you can still get in, but the NOCs at local utilities and telcos are notified, maybe police as well. If you can tie access to a particular person's ID, I doubt that person will misuse it. Of course, this requires power and battery backup. On the other hand, maybe it's time to put the blame on the unions. If the saboteur is found to be a union member, maybe penalize the entire union somehow, since they're acting like a terrorist group at that point. Chuck -Original Message- From: Lamar Owen [mailto:lo...@pari.edu] Sent: Monday, April 13, 2009 11:22 AM To: nanog@nanog.org Subject: Re: Cart and Horse On Monday 13 April 2009 11:06:55 Roy wrote: A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims. :-) Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all. PGP.sig Description: This is a digitally signed message part
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Dylan Ebner wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. http://swm.pp.se/bayarea.jpg This is the bay area as well... How long do you need to spend with a torch to cut thru that? A couple of minutes? There is absolutely no way you can stop a determined attacker, and it would increase cost a lot more than it's worth. Time is better spent stopping the few people who actually do these kinds of things, same way as it's not worth it for regular people to wear body armour all the time, just in case they might get shot, or have parachutes and emergency exits that work in mid-flight on commercial airliners. The various police agencies and the NTSB cost less in a cost/benefit analysis. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Cart and Horse
On 4/13/09, Lamar Owen lo...@pari.edu wrote: On Monday 13 April 2009 11:06:55 Roy wrote: A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims. :-) Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all. *heh* Just in case the next vandals slice the fiber, then weld the manhole covers shut on the way out? I guess the only thing worse would be for the vandals to have a truckload of quick-drying cement with them; slice the fiber, dump quick-drying cement into the vault, pop the lid on, tamp thermite in the gap around the rim and flash weld it shut. Talk about creating an extended outage scenario. ^_^;
Re: Fiber cut in SF area
It all comes down to money... It will cost them lots of it to get power and some type of readers installed to monitor manhole access... There has always been a lack of security on the telco side, this incident just brings it to light... In my town many of the verizon fios boxes are not locked and the wiring frame boxes for pots line neither.. Its all of a matter of how much cash they wanna throw at it... Sent on the Now Network� from my Sprint® BlackBerry -Original Message- From: Dylan Ebner dylan.eb...@crlmed.com Date: Mon, 13 Apr 2009 09:57:30 To: nanog@nanog.org Subject: RE: Fiber cut in SF area One thing that is missing here is before we can define security we need to define the threat and the obstruction the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them. So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the Threat to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, where do we go from here? It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.eb...@crlmed.com www.consultingradiologists.com -Original Message- From: Joe Greco [mailto:jgr...@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Thanks, *excellent* example. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Of course, but I said if we accept that. It was a challenge for the previous poster. ;-) Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh
Re: Fiber cut in SF area
On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not minimize it. Flag this under the whole obscurity category, but think about this - if you're a vandal itching to do something stupid, and you see a bunch of manhole covers and a couple of them have locks on them, which ones are you going to target? The ones with the locks, of course. Why? Because by the very existence of the locks, it implies there's something of considerable value beyond the lock. -Andy
Re: Fiber cut in SF area
On 4/13/09, Dylan Ebner dylan.eb...@crlmed.com wrote: My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. Actually, in many ways it's getting easier; now, you can sign an NDA with your fiber providers and get GIS data for the fiber runs which you can pop into Google Earth, and verify path separation along the entire run; you put notification requirements into the contract stipulating that the fiber provider *must* notify you and provide updated GIS data if the path must be physically moved, and the move deviates the path by more than 50 feet from the previous GIS data; and you put escape clauses into the contract in case the re-routing of the fiber unavoidably reduces or eliminates your physical run diversity from your other providers. In years past, trying to overlay physical map printouts to validate path separation was a nightmare. Now, standardized GIS data formats make it a breeze. protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. Sometimes a little bit of paranoia can help save your butt...or at least keep you out of the hot seat. Matt
Re: Fiber cut in SF area
I guess the next generation fiber networks will need to be installed with tunnel boring machines and just not surface anywhere except the endpoints :) After all, undersea cables get along just fine without convenient access along their length... On Mon, Apr 13, 2009 at 12:12 PM, Mikael Abrahamsson swm...@swm.pp.sewrote: On Mon, 13 Apr 2009, Dylan Ebner wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. http://swm.pp.se/bayarea.jpg This is the bay area as well... How long do you need to spend with a torch to cut thru that? A couple of minutes? There is absolutely no way you can stop a determined attacker, and it would increase cost a lot more than it's worth. Time is better spent stopping the few people who actually do these kinds of things, same way as it's not worth it for regular people to wear body armour all the time, just in case they might get shot, or have parachutes and emergency exits that work in mid-flight on commercial airliners. The various police agencies and the NTSB cost less in a cost/benefit analysis. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Paetec MPLS + BGP solution opinions
Hi all - I was wondering if anyone could offer any opinions or share some experiences about Paetec, and more specifically their MPLS, BGP, and Network Firewall services. I just started at a new employer and they would like get into a more robust DR strategy involving both our locations and public services. They are suggesting that we use MPLS connections to their bandwidth infrastructure, and make use of their Network Firewall services as a front end for our public services. This way we can make use of their front end BGP without having to qualify for an ARIN allocation. I come from a company where we had our own diverse providers and had an ARIN allocation, so I have not used a managed solution like Paetec is offering. Any experience or comments would be greatly appreciated. Thank you, Jeffrey
Re: Fiber cut in SF area
Or skip the locks and fill the manholes with sand. Then provide the service folks those big suction trucks to remove the sand for servicing :) On Mon, Apr 13, 2009 at 12:28 PM, Andy Ringsmuth andyr...@inebraska.comwrote: On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote: Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. It doesn't stop it, it just makes it slightly harder, and they'll go after another point. IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not minimize it. Flag this under the whole obscurity category, but think about this - if you're a vandal itching to do something stupid, and you see a bunch of manhole covers and a couple of them have locks on them, which ones are you going to target? The ones with the locks, of course. Why? Because by the very existence of the locks, it implies there's something of considerable value beyond the lock. -Andy
Re: Fiber cut in SF area
On Mon, 13 Apr 2009, Dorn Hetzel wrote: I guess the next generation fiber networks will need to be installed with tunnel boring machines and just not surface anywhere except the endpoints :) After all, undersea cables get along just fine without convenient access along their length... Boat anchors and earthquakes do a pretty effective job of cutting submarine cables. jms
Re: Cart and Horse
Church, Charles wrote: Wouldn't some authentication system be more useful than trying to lock all the manholes? Picture a system maybe using RFID or some other radio system where you walk up to manhole, wave your 'wand' (like a Mobil Speedpass), you hear a couple beeps, and you're cleared to open the manhole. Without authenticating, you can still get in, but the NOCs at local utilities and telcos are notified, maybe police as well. If you can tie access to a particular person's ID, I doubt that person will misuse it. Get the guy drunk on Friday night, pickpocket his ID, cut fiber. Roy r.engehau...@gmail.com wrote: A friend mentioned at dinner yesterday that he spotted several ATT trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims. And now the security theater begins. jc
Verizon BGP Contact
Could someone from Verizon contact me off list? We are having some problems with a new turn up with 2 Gig Links, and tech support has not been much help over the last few days in trying to get this resolved. Thanks, Brian
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. You can't keep people out, especially since these manholes and tunnels are designed FOR human access. But a better job can be done of monitoring and knowing what is going on in the tunnels and access points from a remote location. Cheap: light sensor + cell phone = knowing exactly when and where the amount of light in the tunnel changes. Detects unauthorized intrusions. Make sure to detect all visible and IR spectrum, should someone very determined use night vision and IR lights to disable the sensor. Mid-Range: Webcam + cell phone = SEEING what is going on plus everything above. High-end: Webcam + cell phone + wifi or wimax backup both watching the entrance and the tunnels. James Bond: Lasers. Active monitoring of each site makes sure each one is online. Pros: * Knowing immediately that there is a change in environment in your tunnels. * Knowing who or at least THAT something is in there * Being able to proactively mitigate attempts * Availability of Arduino, SIM card adapters, and sophisticated sensor and camera equipment at low cost Cons: * Cell provider outage or spectrum blocker removes live notifications * False positives are problematic and can lower monitoring thresholds * Initial expense of deployment of monitoring systems Farmers use tiny embedded devices on their farms to monitor moisture, rain, etc. in multiple locations to customize irrigation and to help avoid loss of crops. These devices communicate with themselves, eventually getting back to a main listening post which relays the information to the farmer's computers. Tiny, embedded, networked devices that monitor the environment in the tunnels that run our fiber to help avoid loss of critical communications services seems to be a good idea. Cheap, disposable devices that can communicate with each other as well as back to some HQ is a way to at least know about problems of access before they happen. No keys to lose, no technology keeping people out and causing repair problems. Some other things that could detect access problems: * Pressure sensors (maybe an open manhole causes a detectable change in air pressure in the tunnel) * Temperature sensors (placed near access points, detects welding and thermite use) * Audio monitor (can help determine if an alert is just a rat squealing or people talking -- could even be automated to detect certain types of noises) * IR (heat) motion detection, as long as giant rats/rodents aren't a problem * Humidity sensors (sell the data to weatherbug!) One last thought inspired by the guy who posted about pouring quick-set concrete in to slow repair. Get some heavy-duty bags, about 10 feet long and large enough to fill the space in the tunnel. More heavily secure the fiber runs directly around the access space, then inflate two bags on either side of the access point. Easily deflated, these devices also have an electronic device which can notify HQ that they are being deflated or the pressure inside is changing (indicating pushing or manipulation). That way you only need to put these bags at access points, not throughout the whole tunnel. Kinda low-tech, but could be effective. No keys needed, could be inflated/deflated quickly, and you still get notification back to a monitoring point. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: [OT] Re: Fiber cut in SF area
On Sun, Apr 12, 2009 at 03:37:00AM +, Paul Vixie wrote: as long as the west's ideological opponents want terror rather than panic, and also to inflict long term losses rather than short term losses, that's true. in this light you can hopefully understand why bollards to protect internet exchanges against truck bombs are not only penny wise pound foolish (since the manholes a half mile away won't be hardened or monitored or even Of the two physical disaster scenarios, i.e. catastrophic destruction of a peering point or multiple long-line break, which do you think is the less costly -- in both time and treasure -- to remedy? It is acknowledged that the result of either is loss of service, but which is the more survivable event? In light of this, where would you focus your finite mitigation efforts? locked) but also completely wrongheaded (since terrorists need publicity which means they need their victims to be fully able to communicate.) Do you realize that you're putting trust in the sane action of parties who conclude their reasoning process with destruction and murder? -- . ___ ___ . . ___ . \/ |\ |\ \ . _\_ /__ |-\ |-\ \__
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: Peter Beckman [mailto:beck...@angryox.com] wrote: Sent: Monday, April 13, 2009 11:19 AM To: Dylan Ebner Cc: nanog@nanog.org Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. $500,000 to get it built with off-the-shelf components, tested in hostile tunnel environments and functioning. Then $350 per device, which would cover 1000 feet of tunnel, or about $2000 per mile for the devices. I'm not sure how things are powered in the tunnels, so power may need to be run, or the system could run off sealed-gel batteries (easily replaced and cheap, powers device for a year), system can be extremely low power. Add a communication device ($1000) every mile or two (the devices communicate between themselves back to the nearest communications device). Total cost, assuming 3 year life span of the device, is about $3000 per mile for equipment, or $1000 per year for equipment, plus $500 per year per mile for maintenance (batteries, service contracts, etc). Assumes your existing cost of tunnel maintenance can also either replace devices or batteries or both. Add a speedy roomba like RC device in the tunnel with an HD cam and a 10 or 20 mile range between charging stations that can move to the location where an anomaly was detected, and save some money on the per-device cost. It could run on an overhead monorail, or just wheels, depending on the tunnel configuration and moisture content. Add yet another system -- an alarm of sorts -- that goes off upon any anomaly being detected, and goes off after 5 minutes of no detection, to thwart teenagers and people who don't know how sophisticated the monitoring system really is. Put the alarm half way between access points, so it is difficult to get to and disable. Network it all, so that it can be controlled and updated from a certain set of IPs, make sure all changes are authenticated using PKI or certificates, and now you've made it harder to hack. Bonus points -- get a communication device that posts updates via SSL to multiple pre-programmed or random Confickr-type domains to make sure the system continues to be able to communicate in the event of a large outage. Then amortize that out to our bills. Extra credit: would you pay for it? Assuming bills in the hundreds of thousands of dollars per month, maybe to the millions of dollars, and then figure out what an outage costs you according to the SLAs. Then figure out how much a breach and subsequent fiber cut costs you in SLA payouts or credits, multiply by 25%, and that's your budget. If the proposed system is less, why wouldn't you do it? The idea is inspired by the way Google does their datacenters -- use cheap, off-the-shelf hardware, network it together in smart ways, make it energy efficient, ... profit! Anyone want to invest? Maybe I should start the business. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
--- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... scott
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, Scott Weeks wrote: --- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... My estimates are for getting something off the ground, equipment-wise, not operationally. What is the cost of the outages? And if this setup can detect un-reported backhoe activity via accelerometers BEFORE it slices through the cable and you can get someone out to investigate the activity before it gets cut, how much is that worth? And my estimate was for the hardware, not training, etc. I'm guessing existing NOCs can easily incorporate new SNMP traps or other methods of alerts into their system fairly easily. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
Peter Beckman [mailto:beck...@angryox.com] wrote: Sent: Monday, April 13, 2009 11:19 AM To: Dylan Ebner Cc: nanog@nanog.org Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: It will be easier to get more divergence than secure all the manholes in the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? Then amortize that out to our bills. Extra credit: would you pay for it? Chris
RE: Fiber cut in SF area
On 4/13/2009 at 1:12 PM, Peter Beckman beck...@angryox.com wrote: On Mon, 13 Apr 2009, Scott Weeks wrote: --- beck...@angryox.com wrote: I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. - Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... My estimates are for getting something off the ground, equipment-wise, not operationally. What is the cost of the outages? But would alarms prevent any, or what proportion, of these incidents? From what we know of this specific one, would an alarm have stopped the perpetrator(s)? It would have bought the NOC five, ten minutes tops before they got the alarm on the circuit. And in practice would a manhole alarm translate to a call to Homeland Security to have the SEALs descend the site pronto, a police unit to roll by when it has the time, or is it going to be an ATT truck rolling by between calls? I'm guessing number two or three, probably three. So what would it get them in this case. If it doesn't deter these guys, who does it deter? And what are the costs of false alarms? What will the ratio of real alarms to false ones be? Maybe lower-stakes vandals take to popping the edge of manhole covers as a little prank. Or that one that triggers whenever a truck tire hits it right. Or the whole line of them that go off whenever the temperature drops below freezing. Or, what I am absolutely sure will happen, miscommunication between repair crews and the NOC about which ones are being moved or field crews opening them without warning the NOC (or even intra-NOC communication). Will they be a boy who cried wolf?
RIM Mail Admin Contact
Hello, If there is anyone from RIM who would be willing to contact me off list I would be most appreciative. Thank you, -David Prude -- David Prude System Administrator Murphy Durieu (212)618-0320
RE: Fiber cut in SF area
Hi Peter, You wrote: So, let's see. I'm pulling numbers out of my butt here, snip Total cost...is about $3000 per mile for equipment snip It could run on an overhead monorail snip Network it all snip Confickr-type domains to make sure I get the feeling you haven't deployed or operated large networks. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. Joel is right: the thing about the outdoors is there's a lot of it. The cost over time investment of copper and fiber communucations networks, power transmission networks, cable transmission networks is pretty well documented elsewhere. Google around a little for them. The investment is tremendous. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. Then amortize that out to our bills. Extra credit: would you pay for it? Assuming bills in the hundreds of thousands of dollars per month, maybe to the millions of dollars, and then figure out what an outage costs you according to the SLAs. Then figure out how much a breach and subsequent fiber cut costs you in SLA payouts or credits, multiply by 25%, and that's your budget. If the proposed system is less, why wouldn't you do it? SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi-path diversity. Here come the microwave towers! The idea is inspired by the way Google does their datacenters -- use cheap, off-the-shelf hardware, network it together in smart ways, make it energy efficient, ... profit! Works great inside four walls. Anyone want to invest? Maybe I should start the business. Nahh, I already have a web cam on my Smarties orb. What else do I really need? Chris
RE: Fiber cut in SF area
On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: I get the feeling you haven't deployed or operated large networks. Nope. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. It depends on where you want to deploy it and how many miles you want to protect. I was thinking along the lines of $1.5 million for 1000 miles of tunnel, equipment only. It assumes existing maintenance crews would replace sensors that break or go offline, and that those expenses already exist. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. What if you had 5 minutes of advanced notice that something was happening in or near one of your Tunnels that served hundreds of thousands of people and businesses and critical infrastructure? Could you get someone on site to stop it? Maybe. Is it worth it? Maybe. Given my inexperience with large networks, maybe fiber cuts and outages due to vandals, backhoes and other physical disruptions are just what we hear about in the news, and that it isn't worth the expense to monitor for those outages. If so, my idea seems kind of silly. SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi-path diversity. Here come the microwave towers! *laugh* Thank goodness for standardized GIS data. :-) --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: [OT] Re: Fiber cut in SF area
I sense a thread moderation occurring here shortly. valdis.kletni...@vt.edu wrote: On Mon, 13 Apr 2009 14:39:23 EDT, Izaac said: Do you realize that you're putting trust in the sane action of parties who conclude their reasoning process with destruction and murder? And how is that different from a US general plotting destruction and the killing of enemy troops during an offensive? And yet we usually trust our generals and call them sane.
Re: Cart and Horse
This is not such an odd solution. Locks are really easy to break with a screw driver and a hammer which almost everyone has and is easy to carry, but most people aren't going to have or carry a torch or a cutting wheel. After 9/11 a large portion of the man holes in NYC were welded shut to prevent them from being used to hide explosives. On Apr 13, 2009, at 6:10 PM, Joel Esler wrote: Yeah, I would have loved to be on the wall during that conversation: So, how can we lock people out of the manholes? We could put locks on them? No, someone could just cut the locks starts laughing We could weld them shut still laughing pointed eared bossGood idea, do it stops laughing, serious lookReally sir? Yes, make it happen all nervously look at each other Uh, okay...
Re: Fiber cut in SF area
This all implies that the majority of fiber is in tunnels that can be monitored. In my experience, almost none of it is in tunnels. In NYC, it's usually buried in conduits directly under the street, with no access, except through the man holes which are located about every 500 feet. In LA, a large amount of the fiber is direct bored under the streets, with access from hand holes and splice boxes located in the grassy areas between the street and the side walks. Along train tracks, the fiber is buried in conduits which are direct buried in the direct along side the train tracks, with hand holes every 1000 feet or so. In any of these scenarios, especially in the third, where the fiber might run through a rural area with no road access and no cellphone coverage. Simply walk through the woods to the train tracks, put open a hand hole and snip, snip, snip, fiber cut. Shane Ronan On Apr 13, 2009, at 5:54 PM, Peter Beckman wrote: On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote: I get the feeling you haven't deployed or operated large networks. Nope. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. It depends on where you want to deploy it and how many miles you want to protect. I was thinking along the lines of $1.5 million for 1000 miles of tunnel, equipment only. It assumes existing maintenance crews would replace sensors that break or go offline, and that those expenses already exist. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list. What if you had 5 minutes of advanced notice that something was happening in or near one of your Tunnels that served hundreds of thousands of people and businesses and critical infrastructure? Could you get someone on site to stop it? Maybe. Is it worth it? Maybe. Given my inexperience with large networks, maybe fiber cuts and outages due to vandals, backhoes and other physical disruptions are just what we hear about in the news, and that it isn't worth the expense to monitor for those outages. If so, my idea seems kind of silly. SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi- path diversity. Here come the microwave towers! *laugh* Thank goodness for standardized GIS data. :-) --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
RE: Fiber cut in SF area
--- On Mon, 4/13/09, chris.ra...@nokia.com chris.ra...@nokia.com wrote: From: Peter Beckman Subject: RE: Fiber cut in SF area Total cost...is about $3000 per mile for equipment I get the feeling you haven't deployed or operated large networks. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. ATT: 888,000 route miles(1). Verizon: 485,000 route miles(2). If we assume that 1/4 of ATT and Verizon's route-miles are in the US(3), this would mean a capital expense of $666M and $364M respectively, not including any costs incurred for maintenance, monitoring, repair, false positive etc. In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. In the words of Randy Bush, I encourage my competitors to do this. David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com 1) http://www.att.com/gen/press-room?pid=4800cdvn=newsnewsarticleid=26554 2) http://mediumbusiness.verizon.com/about/network.aspx 3) I believe this to be an underestimate.
Re: Fiber cut in SF area
On 14/04/2009, at 11:35 AM, David Barak wrote: In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. Some statistics about the effectiveness of car alarms and unmonitored house alarms would probably be useful here. Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. -- Nathan Ward
Re: Fiber cut in SF area
But that would not be NEBS Complient -PHB I have thought of air horns in my colo cage when a tech of mine messes up. --Original Message-- From: Nathan Ward To: nanog list Subject: Re: Fiber cut in SF area Sent: Apr 13, 2009 4:55 PM On 14/04/2009, at 11:35 AM, David Barak wrote: In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. Some statistics about the effectiveness of car alarms and unmonitored house alarms would probably be useful here. Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. -- Nathan Ward
Re: Fiber cut in SF area
Nathan Ward wrote: Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Most of these measures also presume no shared access. I don't know the layout in the area, but I would expect that some manholes/routes are shared usage and maintenance. Not that my rural self remembers what a manhole looks like under the lid. :) I'm betting inside job, which means redundant routes, security measures, etc all tend to go out the window unless some serious money goes into it, and even then, is there a security mechanism that can't be broken? Jack
Re: Fiber cut in SF area
There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P
RE: Fiber cut in SF area
Though I think networked environmental monitoring has its merits, it's clear the technology is unproven in monitoring fiber tunnels, and my inexperience in running and managing such tunnels makes this thread bordering on off-topic. I'm happy to continue conversations via email, but this will be my last on-list reply regarding the topic I started. On Mon, 13 Apr 2009, Crist Clark wrote: But would alarms prevent any, or what proportion, of these incidents? It's hard to say without researching. Sometimes such research shows amazing results that shock people in the industry. Hospitals were shocked to see surgical mistakes reduced by 80+% after implementing a checklist that both doctors and nurses had to go through prior to starting the procedure, and having the patient also go over and approve what was to be done. The stories you hear of people who are getting amputated writing this leg and X X X NOT THIS LEG before surgery is a result of these studies and checklists. RFID-tagged surgical components and gauze pads are another tech tool being used after such research. You'd think a checklist wouldn't really help, but in reality it made industry changing and life-saving differences. While active alarms and monitoring of fiber tunnels would do the same, but without research, nobody can say for sure how effective or ineffective such a system would be. From what we know of this specific one, would an alarm have stopped the perpetrator(s)? It would have bought the NOC five, ten minutes tops before they got the alarm on the circuit. And in practice would a manhole alarm translate to a call to Homeland Security to have the SEALs descend the site pronto, a police unit to roll by when it has the time, or is it going to be an ATT truck rolling by between calls? I'm guessing number two or three, probably three. So what would it get them in this case. If it doesn't deter these guys, who does it deter? It's not there as a deterrent. It's there to allow a NOC to know that something is going on in a tunnel where potentially critical infrastructure resides. Maybe it doesn't prevent the malicious cut, but combined with video surveilence, it could identify the cutters. Audio recording devices could record voices. I assume large networks have large 24/7 crews. Get a truck to roll (once you sufficiently trust the system) or get a contractor who resides nearby to check out the area. When the alarm goes off, you go check it. If you welded the manholes shut, and there are no scheduled maintenance windows for that area, you can be pretty damn sure something untoward is going on, or it'll be a company truck roll that didn't follow procedure. And what are the costs of false alarms? What will the ratio of real alarms to false ones be? Maybe lower-stakes vandals take to popping the edge of manhole covers as a little prank. Weld 'em shut. Use one of those special screws that you can only unscrew with the right equipment (worked wonders for the tire industry with the lock nut). It won't stop anyone determined, but 13 year olds with M80s will move on. If you get a certain location that continues to get false alarms due to vandals, put in a highpowered webcam to monitor the location. Use ZoneMinder to monitor and record motion. Make sure the camera does nighttime well. Then when you have an alarm, check the video. Or that one that triggers whenever a truck tire hits it right. I would envision that though every device would report the same data with the same sensitivity, false alarms could be mitigated through filters for a given location. Tunnels near train tracks would be filtered differently than tunnels in the middle of a field under high power lines. Or the whole line of them that go off whenever the temperature drops below freezing. The device would go through a lot of environmental testing, so that its upper and lower operating limits could be known. Hardened where necessary. Or, what I am absolutely sure will happen, miscommunication between repair crews and the NOC about which ones are being moved or field crews opening them without warning the NOC (or even intra-NOC communication). Will they be a boy who cried wolf? Maybe. Maybe the whole idea is way too far fetched. Maybe my impression of the state of affairs when it comes to fiber tunnels is really not that big of a deal, and that outages due to physical access (humans, backhoes, floods) don't make up a significant portion of outages, and this is not a problem that fiber companies want to solve. Clearly there are a lot of problems that this sort of monitoring could face. Given sufficient time to mature, I think cheap, repeatable monitoring devices networked together can be a valuable asset, rather than yet another annoying alarm NOC folk and maintenance crews grow to hate and simply not be effective. --- Peter
Re: Fiber cut in SF area
Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Better they cut the fiber instead of Oklahoma Citying the central office.
Re: Fiber cut in SF area
But you are ignoring the cost of designing, procuring, installing, monitoring, maintaining such a solution for the THOUSANDS of man holes and hand holes in even a small fiber network. The reality is, the types of outages that these things would protect against (intentional damage to the physical fiber) just don't happen often enough to warrant the cost. These types of solutions don't protect against back hoes digging up the fiber, as even if they gave a few minutes of advanced notice, the average telco can't get someone to respond to a site in an hour let alone minutes. On Apr 13, 2009, at 9:05 PM, Peter Beckman wrote: On Mon, 13 Apr 2009, Shane Ronan wrote: This all implies that the majority of fiber is in tunnels that can be monitored. In my experience, almost none of it is in tunnels. In NYC, it's usually buried in conduits directly under the street, with no access, except through the man holes which are located about every 500 feet. In LA, a large amount of the fiber is direct bored under the streets, with access from hand holes and splice boxes located in the grassy areas between the street and the side walks. Along train tracks, the fiber is buried in conduits which are direct buried in the direct along side the train tracks, with hand holes every 1000 feet or so. In any of these scenarios, especially in the third, where the fiber might run through a rural area with no road access and no cellphone coverage. Simply walk through the woods to the train tracks, put open a hand hole and snip, snip, snip, fiber cut. I'm sure more malicious fiber cuts would result in heightened security. If you can put your hand in it, you could put a sensor in it. It wouldn't work everywhere, but it could work even in conduit or just simply inside access points. A device the size of your fist or smaller could do the monitoring, and would fit in most access points I would guess. You can't protect it all, and obviously you can't put a camera at every access point (well, maybe you can). You can't stop a determined person from doing anything (like promote networked smart sensors for fiber runs, or setting a small explosion inside an access point). And maybe environmental monitoring of these areas just won't do anything to help. But who knows. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Fiber cut in SF area
On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach writes: protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. This completely ignores the grooming problem. Not completely; it just gives you teeth for exiting your contract earlier and finding a more responsible provider to go with who won't violate the terms of the contract and re-groom you without proper notification. I'll admit I'm somewhat simplifying the scenario, in that I also insist on no single point of failure, so even an entire site going dark doesn't completely knock out service; those who have been around since the early days will remember my email to NANOG about the gas main cut in Santa Clara that knocked a good chunk of the area's connectivity out, *not* because the fiber was damaged, but because the fire marshall insisted that all active electrical devices be powered off (including all UPSes) until the gas in the area had dissipated. Ever since then, I've just acknowledged you can't keep a single site always up and running; there *will* be events that require it to be powered down, and part of my planning process accounts for that, as much as possible, via BCP planning. Now, I'll be the first to admit it's a different game if you're providing last-mile access to single-homed customers. But sitting on the content provider side of the fence, it's entirely possible to build your infrastructure such that having 3 or more OC192s cut at random places has no impact on your ability to carry traffic and continue functioning. You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business. You simply count on them not playing entirely fair, and penalize them when they don't; and you have enough parallel contracts with different providers at different sites that outages don't take you completely offline.
Re: Fiber cut in SF area
On Apr 13, 2009, at 8:31 PM, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. I certainly think this trailer is the most insightful thought of the day. When you're looking for backup comms, is it just going to be the ham radio operators and am/fm radio stations left if there were some outage? With tv having gone digital it's not possible to tune in and pick up the audio carrier anymore. Wartime and times of civil unrest the first thing you do is take over communication to the citizens. Without your internet^Wpodcast of the news, how will you know what is going on? If redundancy is sacrificed in the name of better quarterly earnings is it the right decision? this is not only interesting from a network operators perspective but from a governance perspective as well. I've not done any ham radio stuff for ~15+ years but do keep a shortwave radio around (battery powered of course). The first thing to happen will be the network will be severed. Look at what happened in Burma. Both their internet links were turned off, and not just taking down BGP, but the circuits were unplugged. - jared
Re: Fiber cut in SF area
There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. I certainly think this trailer is the most insightful thought of the day. When you're looking for backup comms, is it just going to be the ham radio operators and am/fm radio stations left if there were some outage? With tv having gone digital it's not possible to tune in and pick up the audio carrier anymore. Wartime and times of civil unrest the first thing you do is take over communication to the citizens. Without your internet^Wpodcast of the news, how will you know what is going on? If redundancy is sacrificed in the name of better quarterly earnings is it the right decision? There is a problem with this thinking, so in case of an emergency you expect to switch and change how you do things?! That will not work, as we can barely make it work under *non_emergency_conditions*. The strategy has too be that things contine to work as they used to do even in an emergency. this is not only interesting from a network operators perspective but from a governance perspective as well. I've not done any ham radio stuff for ~15+ years but do keep a shortwave radio around (battery powered of course). Ham's can do orderwire, but not replace for example a IP network, if you are lucky, you get kilobits on shoer wave with 10e-5 BER.. The first thing to happen will be the network will be severed. Look at what happened in Burma. Both their internet links were turned off, and not just taking down BGP, but the circuits were unplugged. The best netweok is the one that never works right, so you excercise the redundancy all the time.. -P
Re: Fiber cut in SF area
Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: protected rings are a technology of the past. Don't count on your vendor to provide redundancy for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc. This completely ignores the grooming problem. Not completely; it just gives you teeth for exiting your contract earlier and finding a more responsible provider to go with who won't violate the terms of the contract and re-groom you without proper notification. That's a post-facto financial recovery / liability limitation technique, not a high availability / hardening technique... I'll admit I'm somewhat simplifying the scenario, in that I also insist on no single point of failure, so even an entire site going dark doesn't completely knock out service; those who have been around since the early days will remember my email to NANOG about the gas main cut in Santa Clara that knocked a good chunk of the area's connectivity out, *not* because the fiber was damaged, but because the fire marshall insisted that all active electrical devices be powered off (including all UPSes) until the gas in the area had dissipated. Ever since then, I've just acknowledged you can't keep a single site always up and running; there *will* be events that require it to be powered down, and part of my planning process accounts for that, as much as possible, via BCP planning. I was less than a mile away from that, I remember it well. My corner cube even faced in that direction. I heard the noise then the net went poof. One of those Oh, that's not good at all combinations. Now, I'll be the first to admit it's a different game if you're providing last-mile access to single-homed customers. But sitting on the content provider side of the fence, it's entirely possible to build your infrastructure such that having 3 or more OC192s cut at random places has no impact on your ability to carry traffic and continue functioning. You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business. You simply count on them not playing entirely fair, and penalize them when they don't; and you have enough parallel contracts with different providers at different sites that outages don't take you completely offline. The problem with grooming is that in many cases, due to provider consolidation and fiber vendor consolidation and cable swap and so forth, you end up with parallel contracts with different providers at different sites that all end up going through one fiber link anyways. I had (at another site) separate vendors with fiber going northbound and southbound out of the two diverse sites. Both directions from both sites got groomed without notification. Slightly later, the northbound fiber was Then rerouted a bit up the road, into a southbound bundle (same one as our now-groomed southbound link), south to another datacenter then north again via another path. To improve route reduncancy northbound overall, for the providers' overall customer links. And the shared link south of us was what got backhoed. This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. -george william herbert gherb...@retro.com
Re: Fiber cut in SF area
On Tue, Apr 14, 2009 at 03:41:25AM +0200, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P Yo, Peter. You speak of infrastructure as if it was a monolithic thing. Why would you think that some localized NoCal fiber cuts would be taking out the whole countrys infrastructure? --bill
Re: Fiber cut in SF area
On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: [much material snipped in the interests of saving precious electron resources...] This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. Ultimately, I think a .sig line I saw years back summed it up very succinctly: Earth is a single point of failure. Below that, you're right, we're all just quibbling about which digits to put to the right of the decimal point. If the entire west coast of the US drops into the ocean, yes, having my data backed up on different continents will help; but I'll be swimming with the sharks at that point, and won't really be able to care much, so the extent of my disaster planning tends to peter out around the point where entire states disappear, and most definitely doesn't even wander into the realm of entire continents getting cut off, or the planet getting incinerated in a massive solar flare. Fundamentally, though, I think it's actually good we have outages periodically; they help keep us employed. When networks run too smoothly, management tends to look upon us as unnecessary overhead that can be trimmed back during the next round of layoffs. The more they realize we're the only bulwark against the impending forces of chaos you mentioned above, the less likely they are to trim us off the payroll. Matt Note--tongue was firmly planted in cheek; no slight was intended against those who may have lost jobs recently; post was intended for humourous consumption only; any resemblence to useful content was purely coincidental and not condoned by any present or past employer. Repeated exposure may be habit forming. Do not read while operating heavy machinery.
Re: Fiber cut in SF area
Rofl Matt, I was recently laid off from my job for 'economic' reasons, what you say is deadly accurate. Bravo! :) On Mon, Apr 13, 2009 at 7:01 PM, Matthew Petach mpet...@netflight.comwrote: On 4/13/09, George William Herbert gherb...@retro.com wrote: Matthew Petach wrote: George William Herbert gherb...@retro.com wrote: Matthew Petach writes: [much material snipped in the interests of saving precious electron resources...] This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc) There may be 4 or 5 nines, but the 1 at the end has your name on it. Ultimately, I think a .sig line I saw years back summed it up very succinctly: Earth is a single point of failure. Below that, you're right, we're all just quibbling about which digits to put to the right of the decimal point. If the entire west coast of the US drops into the ocean, yes, having my data backed up on different continents will help; but I'll be swimming with the sharks at that point, and won't really be able to care much, so the extent of my disaster planning tends to peter out around the point where entire states disappear, and most definitely doesn't even wander into the realm of entire continents getting cut off, or the planet getting incinerated in a massive solar flare. Fundamentally, though, I think it's actually good we have outages periodically; they help keep us employed. When networks run too smoothly, management tends to look upon us as unnecessary overhead that can be trimmed back during the next round of layoffs. The more they realize we're the only bulwark against the impending forces of chaos you mentioned above, the less likely they are to trim us off the payroll. Matt Note--tongue was firmly planted in cheek; no slight was intended against those who may have lost jobs recently; post was intended for humourous consumption only; any resemblence to useful content was purely coincidental and not condoned by any present or past employer. Repeated exposure may be habit forming. Do not read while operating heavy machinery. -- Respectfully, Chris Hart George Carlinhttp://www.brainyquote.com/quotes/authors/g/george_carlin.html - Frisbeetarianism is the belief that when you die, your soul goes up on the roof and gets stu...
Re: Fiber cut in SF area
On Tue, Apr 14, 2009 at 03:41:25AM +0200, Peter Lothberg wrote: There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P Yo, Peter. You speak of infrastructure as if it was a monolithic thing. Why would you think that some localized NoCal fiber cuts would be taking out the whole countrys infrastructure? --bill If you are talking residential access, in the future when people work from home, the study we did in 2000 came down to that you can only loose 30 subs on a single-point-of failure tehing, and the recomendation was to interlave them, so your neighbour would have connectivity. While on this, we have an even bigger problem, the impact of loosing power is bigger, but their system has not gained the same amount of complexity as ours in the last 100 years.. (the book from 1907 on power-lines is still applicable.) -P
Re: Fiber cut in SF area
telmn...@757.org wrote: Presumes the perp isn't familiar with the hole, and it's security measures. In this case, I doubt that either is the case. Pop in, snip the wires on the horn, and do what you do. Better they cut the fiber instead of Oklahoma Citying the central office. If you're referring to the Event, that scares me every day about the largest meet points in the nation and how much traffic can really fully switch to other paths should one or two disappear completely. On the data side of things, though, while it still takes time, I'm forever impressed at how fast everything comes together to get communications rolling again. Man-made or natural, disasters bring out the best and the worst. Of course, I mostly see natural disasters; wasn't far from the tornado that decorated the Tandy building in Ft. Worth, was 5 miles from the Tornado in Moore, OK, and was bunkered down in my house in Lone Grove this year. I've seen 2 man-made disasters and 2 natural disasters so far this year. One was severe at a network level (Building power outage because the NOC chose not to check it out and discover the faulty power transfer switch; batteries died 8 hours later), and 3 were local and only effected a subset of end users due to cable damage (Tornado in Lone Grove back in Feb, wildfires last week in Lone Grove, and one of our nearby towns had an oversized truck grab the overhead cable and drag it down the road, ripping poles out of the ground, and of course he didn't stick around to pay the bill). If you're referring to our infrastructure, no comment but lots of laughter. I really haven't considered the SF fiber cut to be a big deal. It may effect more people, but it's still a couple minor cuts. From the back woods, Jack
Re: Fiber cut in SF area
On Apr 13, 2009, at 8:40 PM, telmn...@757.org wrote: Better they cut the fiber instead of Oklahoma Citying the central office. I'm not sure that the someone will alway s find the weakest link argument can be summed up any better than this. If you don't believe it, you all need to spend more time in the big room with the blue ceiling outside of your colos/DCs. Daryl
