subject:"Fiber cut in SF area"

Anyone know where the actual cut is?

On 4/9/09, David W. Hankins  wrote:
> On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
>> Just dropping a note that there is a fiber cut in the SF area (I have a
>> metro line down).  AboveNet is reporting issues and I've heard unconfirmed
>> reports that ATT and VZW are affected as well.
>
> Confirmed VZW & ATT;
>
>   http://cbs5.com/local/phone.internet.outage.2.980578.html
>
> Rather widespread "general telco" outage, the county has deployed
> extra patrol units in the south bay to compensate for not being able
> to call 911.
>
> Third video link in shows repairs underway.
>
> --
> David W. Hankins  "If you don't do it right the first time,
> Software Engineer  you'll just have to do it again."
> Internet Systems Consortium, Inc. -- Jack T. Hankins
>

-- 
Sent from my mobile device

Re: Fiber cut in SF area

2009-04-09 Thread Christopher Morrow

isn't there a mailing list for this sort of thing? outages@ I think it is?

(not that I mind, just a little advert for the appropriate forum, and
a place that MAY have some useful info on this topic)
-chris

On Thu, Apr 9, 2009 at 1:51 PM, Ravi Pina  wrote:
> News coverage:
>
> http://cow.org/r/?5459
> http://cow.org/r/?545a
>
> And not that I expect any useful updates:
>
> http://twitter.com/attnews
>
> -r
>
> On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
>> Just dropping a note that there is a fiber cut in the SF area (I have a
>> metro line down).  AboveNet is reporting issues and I've heard unconfirmed
>> reports that ATT and VZW are affected as well.
>>
>> Rgs,
>> craig
>>
>>
>
>

Re: Fiber cut in SF area




Ravi Pina wrote:

News coverage:

http://cow.org/r/?5459
http://cow.org/r/?545a

And not that I expect any useful updates:

http://twitter.com/attnews




Lots of folks covering the same thing...

http://search.twitter.com/search?q=fiber+cut
http://search.twitter.com/search?q=outage

Also reports of power outages as well:
http://search.twitter.com/search?q=power+outage



Here is something interesting...
http://twist.flaptor.com/trends?gram=outage&table=1&tz=-7
http://twist.flaptor.com/trends?gram=fiber%20cut&table=1&tz=-7

Re: Fiber cut in SF area

2009-04-09 Thread Roy

Service to South Santa Clara county is completely down: Internet,
landline, and cellphones.  Both Verizon and AT&T are affected.  911 is
also down.

My cellphones show one or no bars.   Normally they are all four bars.

The idea that all of that is lumped in one fiber bundle is mind boggling.

On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman  wrote:
> I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still
> down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul
> outages are associated with where this ATM terminates and that's the cause,
> rather than the service in/out of Santa Cruz County, but I have no way of
> telling which from here)
>
> My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise
> there), and I'm not seeing significant routing problems in/out of there with
> transit or peering. (Not even any down peers, so no inter-Equinix-site
> outage apparently).
>
> Matthew Kaufman
> matt...@eeph.com
>
>

Re: Fiber cut in SF area

2009-04-09 Thread Christian Koch

Monterey Highway I think

On Thu, Apr 9, 2009 at 11:11 AM, Mike Lyon  wrote:

> Anyone know where the actual cut is?
>
> On 4/9/09, David W. Hankins  wrote:
> > On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
> >> Just dropping a note that there is a fiber cut in the SF area (I have a
> >> metro line down).  AboveNet is reporting issues and I've heard
> unconfirmed
> >> reports that ATT and VZW are affected as well.
> >
> > Confirmed VZW & ATT;
> >
> >   http://cbs5.com/local/phone.internet.outage.2.980578.html
> >
> > Rather widespread "general telco" outage, the county has deployed
> > extra patrol units in the south bay to compensate for not being able
> > to call 911.
> >
> > Third video link in shows repairs underway.
> >
> > --
> > David W. Hankins  "If you don't do it right the first time,
> > Software Engineer  you'll just have to do it again."
> > Internet Systems Consortium, Inc. -- Jack T. Hankins
> >
>
> --
> Sent from my mobile device
>
>

Re: Fiber cut in SF area

2009-04-09 Thread Ravi Pina

>From the news coverage it appears to be in the general area of
http://cow.org/r/?545c

-r


On Thu, Apr 09, 2009 at 11:11:58AM -0700, Mike Lyon wrote:
> Anyone know where the actual cut is?
> 
> On 4/9/09, David W. Hankins  wrote:
> > On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
> >> Just dropping a note that there is a fiber cut in the SF area (I have a
> >> metro line down).  AboveNet is reporting issues and I've heard unconfirmed
> >> reports that ATT and VZW are affected as well.
> >
> > Confirmed VZW & ATT;
> >
> > http://cbs5.com/local/phone.internet.outage.2.980578.html
> >
> > Rather widespread "general telco" outage, the county has deployed
> > extra patrol units in the south bay to compensate for not being able
> > to call 911.
> >
> > Third video link in shows repairs underway.
> >
> > --
> > David W. Hankins"If you don't do it right the first time,
> > Software Engineeryou'll just have to do it again."
> > Internet Systems Consortium, Inc.   -- Jack T. Hankins
> >
> 
> -- 
> Sent from my mobile device

Re: Fiber cut in SF area

2009-04-09 Thread Chris Cariffe

Monterey Road just north of Blossom Hill, San Jose

On Thu, Apr 9, 2009 at 11:11 AM, Mike Lyon  wrote:
> Anyone know where the actual cut is?
>
> On 4/9/09, David W. Hankins  wrote:
>> On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
>>> Just dropping a note that there is a fiber cut in the SF area (I have a
>>> metro line down).  AboveNet is reporting issues and I've heard unconfirmed
>>> reports that ATT and VZW are affected as well.
>>
>> Confirmed VZW & ATT;
>>
>>       http://cbs5.com/local/phone.internet.outage.2.980578.html
>>
>> Rather widespread "general telco" outage, the county has deployed
>> extra patrol units in the south bay to compensate for not being able
>> to call 911.
>>
>> Third video link in shows repairs underway.
>>
>> --
>> David W. Hankins      "If you don't do it right the first time,
>> Software Engineer                  you'll just have to do it again."
>> Internet Systems Consortium, Inc.             -- Jack T. Hankins
>>
>
> --
> Sent from my mobile device
>
>

Re: Fiber cut in SF area

2009-04-09 Thread George William Herbert


Mike Lyon writes:
>Anyone know where the actual cut is?

According to SF Chronicle:
http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/04/09/BAP816VTE6.DTL&tsp=1

"The fiber-optic cables were severed shortly before 1:30 a.m. along Monterey
Highway north of Blossom Hill Road in south San Jose, police Sgt. Ronnie Lopez
said."

Vicintity of 121.81 W 37.26 N, but I have no idea specifically where in that
general area.  There are train tracks through there, could well be
a vault along the train tracks alongside Monterey Highway.  But I don't
know specifically where the AT&T fiber runs down there.


-george william herbert
gherb...@retro.com

Re: Fiber cut in SF area

2009-04-09 Thread Jorge Amodio

On Thu, Apr 9, 2009 at 1:20 PM, Christopher Morrow
 wrote:
> isn't there a mailing list for this sort of thing? outages@ I think it is?

Jared put together long time ago  seems to still be
active and receiving reports about this one.

List archive is at https://puck.nether.net/pipermail/outages/

My .02
Jorge

Re: Fiber cut in SF area

2009-04-09 Thread Alex H. Ryu

Hey Chris,

Yes. outa...@outages.org is the one.

Alex


Christopher Morrow wrote:
> isn't there a mailing list for this sort of thing? outages@ I think it is?
>
> (not that I mind, just a little advert for the appropriate forum, and
> a place that MAY have some useful info on this topic)
> -chris
>
> On Thu, Apr 9, 2009 at 1:51 PM, Ravi Pina  wrote:
>   
>> News coverage:
>>
>> http://cow.org/r/?5459
>> http://cow.org/r/?545a
>>
>> And not that I expect any useful updates:
>>
>> http://twitter.com/attnews
>>
>> -r
>>
>> On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:
>> 
>>> Just dropping a note that there is a fiber cut in the SF area (I have a
>>> metro line down). ㅤㅈㅐㄽboveNet is reporting issues and I've heard unconfirmed
>>> reports that ATT and VZW are affected as well.
>>>
>>> Rgs,
>>> craig
>>>
>>>
>>>   
>> 
>
>
>
>
>

Re: Fiber cut in SF area


Yeah. It's on outages. Not much useful there.

Christopher Morrow wrote:

isn't there a mailing list for this sort of thing? outages@ I think it is?

(not that I mind, just a little advert for the appropriate forum, and
a place that MAY have some useful info on this topic)
-chris

On Thu, Apr 9, 2009 at 1:51 PM, Ravi Pina  wrote:

News coverage:

http://cow.org/r/?5459
http://cow.org/r/?545a

And not that I expect any useful updates:

http://twitter.com/attnews

-r

On Thu, Apr 09, 2009 at 08:14:15AM -0700, Craig Holland wrote:

Just dropping a note that there is a fiber cut in the SF area (I have a
metro line down).  AboveNet is reporting issues and I've heard unconfirmed
reports that ATT and VZW are affected as well.

Rgs,
craig

Re: Fiber cut in SF area

2009-04-09 Thread Christian Koch

nice article on bitgravity blog regarding the cuts..

http://sandbox.bitgravity.com/blog/2009/04/09/destroy-the-internet-with-a-hacksaw/



On Thu, Apr 9, 2009 at 11:22 AM, Charles Wyble wrote:

>
>
> Ravi Pina wrote:
>
>> News coverage:
>>
>> http://cow.org/r/?5459
>> http://cow.org/r/?545a
>>
>> And not that I expect any useful updates:
>>
>> http://twitter.com/attnews
>>
>>
>
> Lots of folks covering the same thing...
>
> http://search.twitter.com/search?q=fiber+cut
> http://search.twitter.com/search?q=outage
>
> Also reports of power outages as well:
> http://search.twitter.com/search?q=power+outage
>
>
>
> Here is something interesting...
> http://twist.flaptor.com/trends?gram=outage&table=1&tz=-7
> http://twist.flaptor.com/trends?gram=fiber%20cut&table=1&tz=-7
>
>
>

Re: Fiber cut in SF area

2009-04-09 Thread Michael Holstein




Anyone know where the actual cut is?

  


Based on the previously posted news articles ..

First one is in this proximity :  37°15'20.79"N 121°48'9.38"W
Second one is in this proximity :  37°29'44.00"N 122°14'44.31"W

First one is along a highway .. second one is along railroad tracks. 
Google Earth's imagray of both areas is quite good (~.5m maybe) .. but 
not quite good enough to make out manholes.


Also interesting to note .. from one of the news articles .. "AT&T's 
contract with the Communication Workers of America expired at 11:59 p.m. 
Saturday"


Cheers,

Michael Holstein
Cleveland State University

Re: Fiber cut in SF area

2009-04-09 Thread George William Herbert


I had written in a NANOG reply:
>Mike Lyon writes:
>Anyone know where the actual cut is?
>
>According to SF Chronicle:
>http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/04/09/BAP816VTE6.DTL&tsp=1
>
>"The fiber-optic cables were severed shortly before 1:30 a.m. along Monterey
>Highway north of Blossom Hill Road in south San Jose, police Sgt. Ronnie Lopez
>said."
>
>Vicintity of 121.81 W 37.26 N, but I have no idea specifically where in that
>general area.  There are train tracks through there, could well be
>a vault along the train tracks alongside Monterey Highway.  But I don't
>know specifically where the AT&T fiber runs down there.

Additional news stories reporting second fiber cut on Sprint fiber
in San Carlos, between San Francisco and San Jose, the SF Gate article above
was updated at 12:20pm with that information.

San Jose cut at around 1:30am, San Carlos around 3:30am.


-george

Re: Fiber cut in SF area

2009-04-09 Thread Jorge Amodio

> My cellphones show one or no bars.   Normally they are all four bars.

hmmm, probably not related but could be that some cellphone operators are
restricting coverage to give priority to emergency svcs communications.

Re: Fiber cut in SF area

2009-04-09 Thread Robert M. Enger



That AT&T has stopped provisioning protection fiber for automatic 
restoral is mind boggling.


That our crack (or on crack) govt contracting/emergency-preparedness 
staff didn't demand protected facilities for 911 is another mind 
boggling issue.


That there is no over-under wide-area back-up coverage for the cellular 
canopy ...


We posture and orate about being prepared for terrorist attacks and 
natural disasters, and then events like these reveal the reality:

   The emperor has no clothes.



Roy wrote:

Service to South Santa Clara county is completely down: Internet,
landline, and cellphones.  Both Verizon and AT&T are affected.  911 is
also down.

My cellphones show one or no bars.   Normally they are all four bars.

The idea that all of that is lumped in one fiber bundle is mind boggling.

On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman  wrote:
  

I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still
down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul
outages are associated with where this ATM terminates and that's the cause,
rather than the service in/out of Santa Cruz County, but I have no way of
telling which from here)

My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise
there), and I'm not seeing significant routing problems in/out of there with
transit or peering. (Not even any down peers, so no inter-Equinix-site
outage apparently).

Matthew Kaufman
matt...@eeph.com

Re: Fiber cut in SF area

2009-04-09 Thread Matthew Kaufman


Robert M. Enger wrote:


We posture and orate about being prepared for terrorist attacks and 
natural disasters, and then events like these reveal the reality:

   The emperor has no clothes.


You wouldn't have clothes either if you could double your profit by not 
wearing any.


Matthew Kaufman

Re: Fiber cut in SF area

Yeah, that's about the right amount of time to crawl out of a man hole,
cover it back up, get in the car, drive to a 24 hour starbucks, pick up some
coffee and drive up to San Carlos, open man-hole, repeat process...



On Thu, Apr 9, 2009 at 12:31 PM, George William Herbert
wrote:

>
> I had written in a NANOG reply:
> >Mike Lyon writes:
> >Anyone know where the actual cut is?
> >
> >According to SF Chronicle:
> >
> http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/04/09/BAP816VTE6.DTL&tsp=1
> >
> >"The fiber-optic cables were severed shortly before 1:30 a.m. along
> Monterey
> >Highway north of Blossom Hill Road in south San Jose, police Sgt. Ronnie
> Lopez
> >said."
> >
> >Vicintity of 121.81 W 37.26 N, but I have no idea specifically where in
> that
> >general area.  There are train tracks through there, could well be
> >a vault along the train tracks alongside Monterey Highway.  But I don't
> >know specifically where the AT&T fiber runs down there.
>
> Additional news stories reporting second fiber cut on Sprint fiber
> in San Carlos, between San Francisco and San Jose, the SF Gate article
> above
> was updated at 12:20pm with that information.
>
> San Jose cut at around 1:30am, San Carlos around 3:30am.
>
>
> -george
>
>

Re: Fiber cut in SF area

2009-04-09 Thread David Edwards


At 12:55 PM 4/9/2009, you wrote:

>From the news coverage it appears to be in the general area of
http://cow.org/r/?545c

-r


Interesting.  The report I got from a vendor was that it is Above.net 
with a fiber cut in Redwood City which is affecting a circuit of mine 
between 200 Paul in SF and PAIX in Palo Alto, which is a ways from 
south San Jose.


David

Re: Fiber cut in SF area

2009-04-09 Thread Gadi Evron


Jorge Amodio wrote:

On Thu, Apr 9, 2009 at 1:20 PM, Christopher Morrow
 wrote:

isn't there a mailing list for this sort of thing? outages@ I think it is?


Jared put together long time ago  seems to still be
active and receiving reports about this one.


Virenda Rode started the outages mailing list.

He even spent money not insignificant buying the outages.org domain from 
someone who owned it.


Gadi.

Re: Fiber cut in SF area

2009-04-09 Thread John A. Kilpatrick


On Thu, 9 Apr 2009, George William Herbert wrote:


"The fiber-optic cables were severed shortly before 1:30 a.m. along Monterey
Highway north of Blossom Hill Road in south San Jose, police Sgt. Ronnie Lopez
said."


The fact that it's vandalism is VERY annoying. Sadly it also shows how 
vulnerable we are.  I'm guessing the next Die Hard movie will have the 
baddies cutting fiber trunks before trying to steal the money?


--
   John A. Kilpatrick
j...@hypergeek.netEmail| http://www.hypergeek.net/
john-p...@hypergeek.net  Text pages|  ICQ: 19147504
 remember:  no obstacles/only challenges

Re: Fiber cut in SF area

2009-04-09 Thread Jeffrey Ollie

On Thu, Apr 9, 2009 at 2:44 PM, Michael Holstein
 wrote:
>
> First one is in this proximity :  37°15'20.79"N 121°48'9.38"W

Street view shows a few manholes in the vicinity.

> Second one is in this proximity :  37°29'44.00"N 122°14'44.31"W

Didn't see anything obvious here.

-- 
Jeff Ollie

Re: Fiber cut in SF area

2009-04-09 Thread JC Dill


Michael Holstein wrote:



Anyone know where the actual cut is?

  


Based on the previously posted news articles ..

First one is in this proximity :  37°15'20.79"N 121°48'9.38"W
Second one is in this proximity :  37°29'44.00"N 122°14'44.31"W

First one is along a highway .. second one is along railroad tracks. 
Google Earth's imagray of both areas is quite good (~.5m maybe) .. but 
not quite good enough to make out manholes.
The manholes are clearly visible on the zoomed in images for the first 
location (Old County Road, San Carlos).  There is a line of 3 closely 
spaced manholes easily seen in the middle of the south/east bound lane - 
about 100-150 feet south/east (towards Redwood City) from where Google 
places this location:


37°29'44.00"N 122°14'44.31"W

If you go to Google's Street View at the second location:

37°29'44.00"N 122°14'44.31"W

There's a manhole right there.  However, the TV footage shows them 
accessing the lines from a manhole in the dirt along the tracks, not a 
manhole in the street.


jc




Also interesting to note .. from one of the news articles .. "AT&T's 
contract with the Communication Workers of America expired at 11:59 
p.m. Saturday"


Cheers,

Michael Holstein
Cleveland State University

RE: Fiber cut in SF area

2009-04-09 Thread Murphy, Jay, DOH

A sobering touché.

Jay Murphy 
IP Network Specialist 
NM Department of Health 
ITSD - IP Network Operations 
Santa Fe, New Mexico 87502 
Bus. Ph.: 505.827.2851

"We move the information that moves your world." 

-Original Message-
From: Robert M. Enger [mailto:en...@enger.us] 
Sent: Thursday, April 09, 2009 1:59 PM
To: Roy
Cc: nanog@nanog.org
Subject: Re: Fiber cut in SF area

That AT&T has stopped provisioning protection fiber for automatic 
restoral is mind boggling.

That our crack (or on crack) govt contracting/emergency-preparedness 
staff didn't demand protected facilities for 911 is another mind 
boggling issue.

That there is no over-under wide-area back-up coverage for the cellular 
canopy ...

We posture and orate about being prepared for terrorist attacks and 
natural disasters, and then events like these reveal the reality:
The emperor has no clothes.

Roy wrote:
> Service to South Santa Clara county is completely down: Internet,
> landline, and cellphones.  Both Verizon and AT&T are affected.  911 is
> also down.
>
> My cellphones show one or no bars.   Normally they are all four bars.
>
> The idea that all of that is lumped in one fiber bundle is mind boggling.
>
> On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman  wrote:
>   
>> I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still
>> down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 Paul
>> outages are associated with where this ATM terminates and that's the cause,
>> rather than the service in/out of Santa Cruz County, but I have no way of
>> telling which from here)
>>
>> My own Gatespeed.net microwave to Equinix SV-3 is working fine (no surprise
>> there), and I'm not seeing significant routing problems in/out of there with
>> transit or peering. (Not even any down peers, so no inter-Equinix-site
>> outage apparently).
>>
>> Matthew Kaufman
>> matt...@eeph.com
>>
>>
>> 
>
>   

__
This inbound email has been scanned by the MessageLabs Email Security System.
__

Confidentiality Notice: This e-mail, including all attachments is for the sole 
use of the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, use, disclosure or distribution is 
prohibited unless specifically provided under the New Mexico Inspection of 
Public Records Act. If you are not the intended recipient, please contact the 
sender and destroy all copies of this message. -- This email has been scanned 
by the Sybari - Antigen Email System.

Re: Fiber cut in SF area

2009-04-09 Thread Jared Mauch



On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:



That AT&T has stopped provisioning protection fiber for automatic  
restoral is mind boggling.


That our crack (or on crack) govt contracting/emergency-preparedness  
staff didn't demand protected facilities for 911 is another mind  
boggling issue.


	This costs $$$ and usually isn't a problem as there are other ways to  
communicate.  The law-enforcement folks qualify for GETS so get  
priority on wired/PSTN.  They can also get radio priority w/ WPS.




That there is no over-under wide-area back-up coverage for the  
cellular canopy ...




	The problem is how do you back up such a large area.  WPS can get you  
priority.


We posture and orate about being prepared for terrorist attacks and  
natural disasters, and then events like these reveal the reality:

  The emperor has no clothes.



	I think the problem is there are clothes, some people/areas have  
none, some have an abundance.  If people don't plan for going out in  
public, there is a chance you'll walk outside naked ;)





Roy wrote:

Service to South Santa Clara county is completely down: Internet,
landline, and cellphones.  Both Verizon and AT&T are affected.  911  
is

also down.

My cellphones show one or no bars.   Normally they are all four bars.




	If you're an ISP, you may be able to obtain GETS or WPS for your  
engineers.  This would allow you a better chance of getting a channel  
to respond to issues.  This is a good test to see how your backup  
plans might work for communication in the case of a larger distaster  
(earthquake, or other).


	Make sure you test the tools you have.  The people I know with GETS  
cards are encouraged to test them regularly and verify they work.  If  
someone has one, I'd be interested to know if it proved to be of value  
today.


- Jared

Re: Fiber cut in SF area



Yep it leads to:




Activity Type Code Desc: PROGRESS COMMENTS
Activity Type Code: PROG

OTDR readings were taken by AT&T West and a cut was located 1600 ft from
the San Jose, CA central office. AT&T West technicians are onsite
working to isolate the exact location of the cut. There are 4 cables
impacted. AT&T Mobility has 61 GSM and 45 co-located UMTS sites out of
service off of Santa Clara Base Station Controllers 15 & 23, and Santa
Clara Radio Network Controller 4. E911 has 52 Location Measuring Units
down. The AT&T West Santa Cruz 11 central office (41,803 ATNs) is
experiencing an SS7 isolation and the San Martin central office (11,904
ATNs) lost it's umbilical and is isolated at this time. The Bailey
remote site (4,973 ATNs) is also isolated. Scott's Valley has 3 out of 4
SS7 links down. The Santa Cruz 01, Aptos, Scott's Valley, Felton,
Boulder Creek, Ben Lomand, San Jose 11, San Jose 13, San Jose 21 central
offices have trunks impacted such that all lines are busy and incoming
calls are receiving trouble messages. The Santa Cruz County SO (178,040
ATNs), Scott's Valley PD (12,007 ATNs) and the UC Santa Cruz PD (14,909
ATNs) are all without ALI at this time. The Gilroy PD PSAP and the
Morgan Hill PD and CDF have been rerouted with ALI/ANI. The Felton CDF
has not been rerouted. There are 17 DSLAMS and 4 ATMS out of service
impacting DSL service. There are 3 SMDI Links down impacting voicemail
service. Verizon's Morgan Hill and Gilroy central offices are currently
isolated. There have been 224,865 blocked calls.



Robert M. Enger wrote:


That AT&T has stopped provisioning protection fiber for automatic 
restoral is mind boggling.


That our crack (or on crack) govt contracting/emergency-preparedness 
staff didn't demand protected facilities for 911 is another mind 
boggling issue.


That there is no over-under wide-area back-up coverage for the cellular 
canopy ...


We posture and orate about being prepared for terrorist attacks and 
natural disasters, and then events like these reveal the reality:

   The emperor has no clothes.



Roy wrote:

Service to South Santa Clara county is completely down: Internet,
landline, and cellphones.  Both Verizon and AT&T are affected.  911 is
also down.

My cellphones show one or no bars.   Normally they are all four bars.

The idea that all of that is lumped in one fiber bundle is mind boggling.

On Thu, Apr 9, 2009 at 11:05 AM, Matthew Kaufman  
wrote:
 

I saw my Sonic.net-over-AT&T ADSL go dark at 02:30 local and it is still
down, served on a fiber remote out of SNCZCA01. (I'm guessing the 200 
Paul
outages are associated with where this ATM terminates and that's the 
cause,
rather than the service in/out of Santa Cruz County, but I have no 
way of

telling which from here)

My own Gatespeed.net microwave to Equinix SV-3 is working fine (no 
surprise
there), and I'm not seeing significant routing problems in/out of 
there with

transit or peering. (Not even any down peers, so no inter-Equinix-site
outage apparently).

Matthew Kaufman
matt...@eeph.com

Re: Fiber cut in SF area

There were multiple cuts. South san jose and san carlos. Yours would
be the san carlos one :)

On 4/9/09, David Edwards  wrote:
> At 12:55 PM 4/9/2009, you wrote:
>> >From the news coverage it appears to be in the general area of
>>http://cow.org/r/?545c
>>
>>-r
>
> Interesting.  The report I got from a vendor was that it is Above.net
> with a fiber cut in Redwood City which is affecting a circuit of mine
> between 200 Paul in SF and PAIX in Palo Alto, which is a ways from
> south San Jose.
>
> David
>

-- 
Sent from my mobile device

Re: Fiber cut in SF area

2009-04-09 Thread Joel Jaeggli



David Edwards wrote:
> At 12:55 PM 4/9/2009, you wrote:
>> >From the news coverage it appears to be in the general area of
>> http://cow.org/r/?545c
>>
>> -r
> 
> Interesting.  The report I got from a vendor was that it is Above.net
> with a fiber cut in Redwood City which is affecting a circuit of mine
> between 200 Paul in SF and PAIX in Palo Alto, which is a ways from south
> San Jose.

redwood city and san carlos on the other hand are right next door to
each other.

http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2009/04/09/BAP816VTE6.DTL&tsp=1

> David

Re: Fiber cut in SF area

2009-04-09 Thread JC Dill


Robert M. Enger wrote:


That AT&T has stopped provisioning protection fiber for automatic 
restoral is mind boggling.


That our crack (or on crack) govt contracting/emergency-preparedness 
staff didn't demand protected facilities for 911 is another mind 
boggling issue.


911 centers can work just fine without phones.  They use radio for 
intra-agency communications with police, fire, etc.  There is also a 
large ham radio community who jumps in to help with communications when 
needed (testing, simulations, and real disasters).


The primary problem a 911 center has if the phones go out is that people 
can't *reach* the 911 center if *their* phone lines don't work.


jc

RE: Fiber cut in SF area

2009-04-09 Thread Carlos Alcantar

Looks like our circuit out of 200 paul from abovenet is back up.

-Original Message-
From: David Edwards [mailto:da...@reliablehosting.com] 
Sent: Thursday, April 09, 2009 1:06 PM
To: nanog@nanog.org
Subject: Re: Fiber cut in SF area

At 12:55 PM 4/9/2009, you wrote:
> >From the news coverage it appears to be in the general area of
>http://cow.org/r/?545c
>
>-r

Interesting.  The report I got from a vendor was that it is Above.net 
with a fiber cut in Redwood City which is affecting a circuit of mine 
between 200 Paul in SF and PAIX in Palo Alto, which is a ways from 
south San Jose.

David

Re: Fiber cut in SF area

2009-04-09 Thread Scott Doty


David Edwards wrote:

At 12:55 PM 4/9/2009, you wrote:

>From the news coverage it appears to be in the general area of
http://cow.org/r/?545c

-r


Interesting.  The report I got from a vendor was that it is Above.net 
with a fiber cut in Redwood City which is affecting a circuit of mine 
between 200 Paul in SF and PAIX in Palo Alto, which is a ways from 
south San Jose.

http://www.kcbs.com/Phone-Outage-Likely-Caused-by-Vandals/4174734

''Police say that at 1:20 a.m., four to five fiber optic cables located 
beneath a manhole were cut and severed on Monterey Highway, north of 
Blossom Hill Road.''


''In San Carlos, vandals struck a second time along Old County Road at 
the edge of San Carlos and Redwood City''


I also heard on KCBS:  The cuts were in 4 manholes in San Carlos, and 
they said it was "seven cables".  (Not sure if that means the same 7 
cables were cut 4 times, or what...)


I also heard: there were 4 cables cut in the South SJ manhole.

A lot of comms (incl. 911) are out for Santa Cruz County, as well as 
South Santa Clara Country, including Gilroy and Morgan Hill.


Just now, from their web stream, they refer to this as "an act of sabotage".

On interview was with an "info-worker" in Morgan Hill, and for her, this 
was "the end of the world".


(Personally, I can think of a "MAE-Clueless" episode that was worse than 
this, but that was in the 90's...)


Finally -- and I'm not a lawyer -- I want to note that killing 911 to a 
city can get you tried for murder in California, if someone dies as a 
result, if I understand the law correctly.


Better days,

-Scott

Re: Fiber cut in SF area

2009-04-09 Thread Raul D. Rincon


http://i.gizmodo.com/5205952/att-putting-up-10-reward-for-cable-cutting-vandals

r


On Apr 9, 2009, at 2:00 PM, Jeffrey Ollie wrote:


On Thu, Apr 9, 2009 at 2:44 PM, Michael Holstein
 wrote:


First one is in this proximity :  37°15'20.79"N 121°48'9.38"W


Street view shows a few manholes in the vicinity.


Second one is in this proximity :  37°29'44.00"N 122°14'44.31"W


Didn't see anything obvious here.

--
Jeff Ollie

Re: Fiber cut in SF area




Jared Mauch wrote:


On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:



That AT&T has stopped provisioning protection fiber for automatic 
restoral is mind boggling.


That our crack (or on crack) govt contracting/emergency-preparedness 
staff didn't demand protected facilities for 911 is another mind 
boggling issue.


This costs $$$ and usually isn't a problem as there are other ways 
to communicate.  The law-enforcement folks qualify for GETS so get 
priority on wired/PSTN.  They can also get radio priority w/ WPS.






I didn't know about WPS.

http://policechiefmagazine.org/magazine/index.cfm?fuseaction=display_arch&article_id=839&issue_id=32006 



Interesting stuff.

Re: Fiber cut in SF area

2009-04-09 Thread Ravi Pina

On Thu, Apr 09, 2009 at 02:06:04PM -0600, David Edwards wrote:
> At 12:55 PM 4/9/2009, you wrote:
> >>From the news coverage it appears to be in the general area of
> >http://cow.org/r/?545c
> >
> >-r
> 
> Interesting.  The report I got from a vendor was that it is Above.net 
> with a fiber cut in Redwood City which is affecting a circuit of mine 
> between 200 Paul in SF and PAIX in Palo Alto, which is a ways from 
> south San Jose.
> 

My company is also impacted by an Abovenet fiber cut, but it is unclear
if it is in any way related to the 2 cuts in this thread.

-r

Re: Fiber cut in SF area

2009-04-09 Thread Ben Scott

On Thu, Apr 9, 2009 at 5:29 PM, Jared Mauch  wrote:
>> That our crack (or on crack) govt contracting/emergency-preparedness staff
>> didn't demand protected facilities for 911 is another mind boggling issue.
>
>  This costs $$$ and usually isn't a problem as there are other ways to
> communicate.   The law-enforcement folks qualify for GETS ...

  Which is fine if you're a law-enforcement folk.  Kinda sucks if
you're an ordinary private citizen who tries to dial 911 and gets a
reorder tone.  Which I presume is what is happening, since everybody
is saying 911 is down.  What's the point of having all the emergency
service personnel communicating with each other if they can't get 911
calls in the first place?  (Rhetorical question, I know there are
other ways they can find out about emergencies, but  911 is the big
one.)

  Maybe nostalgia just ain't what it used to be, but I thought the
PSTN used to be more reliable than this.

#ifdef CONSPIRACY_THEORIST

  What if this isn't simple vandalism?

#endif

-- Ben

Re: Fiber cut in SF area

Appears I can get to Yahoo without 4000ms of latency now too and I don't
have to be routed from San Jose, Ca to Philly to DC.

-Mike



On Thu, Apr 9, 2009 at 3:02 PM, Carlos Alcantar  wrote:

> Looks like our circuit out of 200 paul from abovenet is back up.
>
> -Original Message-
> From: David Edwards [mailto:da...@reliablehosting.com]
> Sent: Thursday, April 09, 2009 1:06 PM
> To: nanog@nanog.org
> Subject: Re: Fiber cut in SF area
>
> At 12:55 PM 4/9/2009, you wrote:
> > >From the news coverage it appears to be in the general area of
> >http://cow.org/r/?545c
> >
> >-r
>
> Interesting.  The report I got from a vendor was that it is Above.net
> with a fiber cut in Redwood City which is affecting a circuit of mine
> between 200 Paul in SF and PAIX in Palo Alto, which is a ways from
> south San Jose.
>
> David
>
>
>

Re: Fiber cut in SF area

2009-04-09 Thread Sean Donelan


On Thu, 9 Apr 2009, Jared Mauch wrote:
That AT&T has stopped provisioning protection fiber for automatic restoral 
is mind boggling.


Only helps with N-1 breaks.  Unfortunately, sometimes there are N+1 
breaks.  Check the NANOG archives, I believe there were 5 breaks in one 
day in the 1990's; and even in the last year there have been 2-4 breaks

on some transoceanic cables at the same time.

On the other hand, I've never heard a carrier complain about digging more 
fiber as long as someone is willing to pay for it.  How much more is 
someone willing to pay to get more diversity?  Not willing to pay for it? 
I guess that's an answer too.



That our crack (or on crack) govt contracting/emergency-preparedness staff 
didn't demand protected facilities for 911 is another mind boggling issue.


	This costs $$$ and usually isn't a problem as there are other ways to 
communicate.  The law-enforcement folks qualify for GETS so get priority on 
wired/PSTN.  They can also get radio priority w/ WPS.


If you don't know the acronyms, see www.ncs.gov.

GETS and WPS are good as long as the system is still connected.  TSP 
and SHARES helps when the system becomes disconnected.  Some carriers 
also have mutual aid pacts, and work with members of the mutual aid pact 
with spare facilities.  Its better to sign up ahead time, rather than 
waiting until after the problem happens.


Even though those tools are useful, also work on how to maintain your 
own self-sufficiency until help arrives.  There will always be some 
prioritization of repair efforts.


Although it had a big impact on some of the largest carriers in the 
region, especially for local services; its always interesting to see other 
stuff kept working.  Not everything broke.



	If you're an ISP, you may be able to obtain GETS or WPS for your 
engineers.  This would allow you a better chance of getting a channel to 
respond to issues.  This is a good test to see how your backup plans might 
work for communication in the case of a larger distaster (earthquake, or 
other).


	Make sure you test the tools you have.  The people I know with GETS 
cards are encouraged to test them regularly and verify they work.  If someone 
has one, I'd be interested to know if it proved to be of value today.


It sucked, but its also an opportunity for ISPs to figure out better ways 
to do things.


personal opinions only

Re: Fiber cut in SF area

2009-04-09 Thread jamie rishaw

On Thu, Apr 9, 2009 at 5:52 PM, Ben Scott  wrote:

>
> #ifdef CONSPIRACY_THEORIST
>
>  What if this isn't simple vandalism?
>
> #endif
>

If my read is correct, this is multiple cuts in multiple locations.

To answer the what-if ("What if this isn't simple vandalism?") : It's not.

-jamie

Re: Fiber cut in SF area

2009-04-09 Thread Robert M. Enger

No RF, no WPS.

If all the base stations are knocked out in a region, and there is no
"over" coverage from towers out of the affected region then there are no
channels to which priority access can be allotted.

A potential remedy (at least for conventional cell phones) would be to
scatter back-up cell sites on high-vantage-point locations. Each would
need to be equipped with multiple narrow sectors using high gain
antennas. These lofty sites would form a secondary canopy over the
region (hence the "over/under" naming). Assuming the secondary sites
are hardened, provided with back-up power and trunked with physical
diversity (perhaps one links using 70/80Ghz), they should provide some
additional protection in situations like this. This would provide some
service when primary towers in an entire sub-area are all knocked out.
Who knows, in day to day routine usage they might even fill-in a few
coverage holes that have been lingering in some systems. From the
reports of "zero bars" on cell phones, we can presume no "over" canopy
is in operation in that region.

There are other radio systems, but their scope is limited. Cellular
provides wider availability. Granny can use her Jitterbug to call for
help. Similarly, many business burglar/fire alarm systems use cellular
to transmit alarms to the central station. With terrestrial and radio
alarm reporting knocked out, many businesses will be sitting ducks.

But why waste the money on system improvements. Best to conserve the
funds to pay bonuses to the corporate executives.
No matter how egregious the error or omission, they always walk away
with big checks, and the rest of us waddle away looking for Preparation-H.

Charles Wyble wrote:

Jared Mauch wrote:

On Apr 9, 2009, at 3:58 PM, Robert M. Enger wrote:

That AT&T has stopped provisioning protection fiber for automatic
restoral is mind boggling.

That our crack (or on crack) govt contracting/emergency-preparedness
staff didn't demand protected facilities for 911 is another mind
boggling issue.

This costs $$$ and usually isn't a problem as there are other
ways to communicate. The law-enforcement folks qualify for GETS so
get priority on wired/PSTN. They can also get radio priority w/ WPS.

I didn't know about WPS.

http://policechiefmagazine.org/magazine/index.cfm?fuseaction=display_arch&article_id=839&issue_id=32006

Interesting stuff.

Re: Fiber cut in SF area

2009-04-09 Thread George William Herbert


Scott Doty wrote:
>(Personally, I can think of a "MAE-Clueless" episode that was worse than 
>this, but that was in the 90's...)

The gas main strike out front of the building in Santa Clara?

Or something else?


-george william herbert
gherb...@retro.com

Re: Fiber cut in SF area

2009-04-10 Thread Scott Doty


George William Herbert wrote:

Scott Doty wrote:
  
(Personally, I can think of a "MAE-Clueless" episode that was worse than 
this, but that was in the 90's...)



The gas main strike out front of the building in Santa Clara?

Or something else?


-george william herbert
gherb...@retro.com
  


Hi George,

No, it was when an AS took their full bgp feed & fed it into their igp (which 
used RIP, iirc), which generated (de-aggregated) routes into /24's, which they then 
announced back into bgp...

iirc, part of the chaos than ensued was due to a router bug, so that the routes 
"stuck around" in global views, even after the AS killed their announcements, 
and even after physically disconnecting from their provider.

We told our customers "the Internet is broken, please try again later"...which 
was acceptable back then.  (But I doubt we would get away with just that nowadays... ;-)  
 )

-Scott

Re: Fiber cut in SF area

2009-04-10 Thread Patrick W. Gilmore


On Apr 10, 2009, at 3:41 PM, Scott Doty wrote:

George William Herbert wrote:

Scott Doty wrote:

(Personally, I can think of a "MAE-Clueless" episode that was  
worse than this, but that was in the 90's...)



The gas main strike out front of the building in Santa Clara?

Or something else?


-george william herbert
gherb...@retro.com


No, it was when an AS took their full bgp feed & fed it into their  
igp (which used RIP, iirc), which generated (de-aggregated) routes  
into /24's, which they then announced back into bgp...


That was Vinny Bono of FLIX, the Fat man Little man Internet eXchange,  
as7007.  Happened in 1997, IIRC.  He used a Bay Networks router to  
redistribute BGP on one card into RIPv1 on another card, stripping the  
CIDR notations off each prefix, making them classful, and stripping  
the AS Path.  This means, for instance, 96.0.0.0 was a /8, not a /24.   
It also means   He then re-redistributed RIP into BGP on a third card,  
which then originated each route from as7007.


I have it on most excellent authority (the "Fat man" himself) that  
this was not possible on ciscos.  Wonder if it is now ... ?


Anyway, I did not know people were calling this the "MAE-Clueless"  
incident.  I've always called it the "7007 incident".  In fact, some  
people still have as7007 filtered.



iirc, part of the chaos than ensued was due to a router bug, so that  
the routes "stuck around" in global views, even after the AS killed  
their announcements, and even after physically disconnecting from  
their provider.


That was Sprint, as7007's transit provider.  Sprint only did AS Path  
filtering, and as every single prefix was ^7007$, they all passed the  
filter.


Vinny literally unplugged the router, no power, no fiber, no copper,  
but the prefixes were still bouncing around the 'Net for hours.   
Sprint kept the routes around for a long time as their routers would  
not honor withdrawals - or so the rumors said.  The rumors also  
claimed the IOS version was named "$FOO-sean".  Sean Doran was CTO of  
Sprint's Internet company at the time, and he supposedly specifically  
asked for the 'feature' of ignoring withdrawals to lower CPU on their  
AGS+s.  I have absolutely no way of confirming this as I haven't  
spoken to Sean in years & years, and wouldn't even know where to find  
him any more.


The most interesting rumor I heard is that Sprint had to shut down  
every single router simultaneously to clear the routes out of their  
network.  Personally I think that's probably a bit exaggerated, but  
who knows?



We told our customers "the Internet is broken, please try again  
later"...which was acceptable back then.  (But I doubt we would get  
away with just that nowadays... ;-)   )


Really?  That's what some broadband providers say nearly daily.

--
TTFN,
patrick

RE: Fiber cut in SF area

2009-04-10 Thread Jo¢

 
I'm confussed, but please pardon the ignorance. 
All the data centers we have are at minimum keys to access
data areas. Not that every area of fiber should have such, but
at least should they? Manhole covers "can" be keyed. For those of
you arguing that this is not enough, I would say at least its a start.
Yes if enough time goes by anything can happen, but how can one
argue an ATM machince that has (at times) thousands of dollars stands
out 24/7 without more immediate wealth. Perhaps I am missing
something here, do the Cops stake out those areas? dunno

Just my 2¢

Re: Fiber cut in SF area

2009-04-11 Thread Joel Jaeggli



Jo¢ wrote:
>  
> I'm confussed, but please pardon the ignorance. 
> All the data centers we have are at minimum keys to access
> data areas. Not that every area of fiber should have such, but
> at least should they? Manhole covers "can" be keyed. For those of
> you arguing that this is not enough, I would say at least it’s a start.
> Yes if enough time goes by anything can happen, but how can one
> argue an ATM machince that has (at times) thousands of dollars stands
> out 24/7 without more immediate wealth. Perhaps I am missing
> something here, do the Cops stake out those areas? dunno

The nice thing about the outdoors is how much of it there is.

> Just my 2¢
> 
> 
> 
> 
> 
> 
>

Re: Fiber cut in SF area

2009-04-11 Thread Joe Greco

> JoÂ¢ wrote:
> > I'm confussed, but please pardon the ignorance. 
> > All the data centers we have are at minimum keys to access
> > data areas. Not that every area of fiber should have such, but
> > at least should they? Manhole covers "can" be keyed. For those of
> > you arguing that this is not enough, I would say at least itâs a start.
> > Yes if enough time goes by anything can happen, but how can one
> > argue an ATM machince that has (at times) thousands of dollars stands
> > out 24/7 without more immediate wealth. Perhaps I am missing
> > something here, do the Cops stake out those areas? dunno
> 
> The nice thing about the outdoors is how much of it there is.

Cute, but a lot of people seem to be wondering this, so a better answer
is deserved.

The ATM machine is somewhat protected for the extremely obvious reason 
that it has cash in it, but an ATM is hardly impervious.

http://www.youtube.com/watch?v=4P8WM8ZZDHk

There are all sorts of strategies for attacking ATM's, and being
susceptible to a sledgehammer, crowbar, or truck smashing into the
unit shouldn't be hard to understand.

Most data centers have security that is designed to keep honest people
out of places that they shouldn't be.  Think that "security guard" at 
the front will stop someone from running off with something valuable?
Maybe.  Have you considered following the emergency fire exits instead?
Running out the loading dock?  Etc?

Physical security is extremely difficult, and defending against a
determined, knowledgeable, and appropriately resourced attacker out to
get *you* is a losing battle, every time.

Think about a door.  You can close your bathroom door and set the privacy
lock, but any adult with a solid shoulder can break that door, or with a 
pin (or flathead or whatever your particular knob uses) can stick it in 
and trigger the unlock.  Your front door is more solid, but if it's wood,
and not reinforced, I'll give my steel-toed boots better than even odds
against it.  What?  You have a commercial hollow steel door?  Ok, that 
beats all of that, let me go get my big crowbar, a little bending will
let me win.  Something more solid?  Ram it with a truck.  You got a
freakin' bank vault door?  Explosives, torches, etc.  Fort Knox?  Bring a
large enough army, you'll still get in.

Notice a pattern?  For any given level of protection, countermeasures are
available.  Your house is best "secured" by making changes that make it
appear ordinary and non-attractive.  That means that a burglar is going to
look at your house, say "nah," and move on to your neighbor's house, where
your neighbor left the garage open.

But if I were a burglar and I really wanted in your house?  There's not
that much you could really do to stop me.  It's just a matter of how well
prepared I am, how well I plan.

So.  Now.  Fiber.

Here's the thing, now.  First off, there usually isn't a financial
motivation to attack fiber optic infrastructure.  ATM's get some
protection because without locks, criminals would just open them and
take the cash.  Having locks doesn't stop that, it just makes it harder.
However, the financial incentive for attacking a fiber line is low.
Glass is cheap.  We see attacks against copper because copper is
valuable, and yet we cannot realistically guard the zillions of miles 
of copper that is all around.

Next.  Repair crews need to be able to access the manholes.  This is a
multifaceted problem.  First off, since there are so many manholes to
protect, and there are so many crews who might potentially need to access
them, you're probably stuck with a "standardized key" approach if you
want to lock them.  While this offers some protection against the average
person gaining unauthorized access, it does nothing to prevent "inside
job" attacks (and I'll note that this looks suspiciously like an "inside
job" of some sort).  Further, any locking mechanism can make it more
difficult to gain access when you really need access; some manholes are
not opened for years or even decades at a time.  What happens when the
locks are rusted shut?  Is the mechanism weak enough that it can be
forced open, or is it tolerable to have to wait extra hours while a
crew finds a way to open it?  Speaking of that, a manhole cover is 
typically protecting some hole, accessway, or vault that's made out of
concrete.  Are you going to protect the concrete too?  If not, what
prevents me from simply breaking away the concrete around the manhole
cover rim (admittedly a lot of work) and just discarding the whole
thing?

Wait.  I just want to *break* the cable?  Screw all that.  Get me a
backhoe.  I'll just eyeball the direction I think the cable's going,
and start digging until I snag something.

Start to see the problems?

I'm not saying that security is a bad thing, just a tricky thing.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contac

Re: Fiber cut in SF area

2009-04-11 Thread Chris Adams

Once upon a time, Jo¢  said:
> Yes if enough time goes by anything can happen, but how can one
> argue an ATM machince that has (at times) thousands of dollars stands
> out 24/7 without more immediate wealth. Perhaps I am missing
> something here, do the Cops stake out those areas? dunno

We've had several occasions here where somebody has stolen a backhoe or
front-end loader from a construction site, driven to the nearest ATM,
and loaded the whole ATM into a (usually stolen) truck.

Also, what is the density of outdoor ATMs?  I'm in a suburban area, and
there may be one every mile or two.  How large is the fiber plant?
Miles and miles of continuous fiber, every inch of which is equally
important.  A lot of it here is even on poles, not buried.

-- 
Chris Adams 
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

Re: Fiber cut in SF area

2009-04-11 Thread Florian Weimer

* Joe Greco:

> The ATM machine is somewhat protected for the extremely obvious reason 
> that it has cash in it, but an ATM is hardly impervious.
>
> http://www.youtube.com/watch?v=4P8WM8ZZDHk

Heh.  Once you install ATMs into solid walls, the attacks get a tad
more interesting.  In some places of the world, gas detectors are
almost mandatory because criminals pump gas into the machine, ignite
it, and hope that the explosion blows a hole into the machine without
damaging the money (which seems to work fairly well if you use the
right gas at the right concentration).

Re: Fiber cut in SF area

2009-04-11 Thread Christopher Morrow

On Sat, Apr 11, 2009 at 11:10 AM, Florian Weimer  wrote:
> * Joe Greco:
>
>> The ATM machine is somewhat protected for the extremely obvious reason
>> that it has cash in it, but an ATM is hardly impervious.
>>
>> http://www.youtube.com/watch?v=4P8WM8ZZDHk
>
> Heh.  Once you install ATMs into solid walls, the attacks get a tad
> more interesting.  In some places of the world, gas detectors are
> almost mandatory because criminals pump gas into the machine, ignite
> it, and hope that the explosion blows a hole into the machine without
> damaging the money (which seems to work fairly well if you use the
> right gas at the right concentration).

also, there is the fact that some very large percentage of ATM
machines were installed with the same admin passwd setup. I recall
~1.5 yrs ago some news about this, and that essentially banks send out
the ATM machines with a stock passwd (sometimes the default which is
documented in easily google-able documents) per bank (BoFA uses
passwd123, Citi uses passwd456 )

I'm not sure that the manholes == atm discussion is valid, but in the
end the same thing is prone to happen to the manholes, there isn't
going to be a unique key per manhole, at best it'll be 1/region or
1/manhole-owner. In the end that key is compromised as soon as the
decision is made :(  Also keep in mind that keyed locks don't really
provide much protection, since anyone can order lockpicks over the
interwebs these days, even to states where ownership is apparently
illegal :(

-Chris

Re: Fiber cut in SF area

2009-04-11 Thread Jorge Amodio

The best protecion is good engineering taking advantage of
technologies and architecures
available since long time ago at any of the different network layers.

Why network operators/carriers don't do it ?, it's another issue and
most of the time
is a question of bottom line numbers for which there are no
engineering solutions.

My .02

RE: Fiber cut in SF area

2009-04-11 Thread Roger Marquis


Jo? wrote:

I'm confussed, but please pardon the ignorance.
All the data centers we have are at minimum keys to access
data areas. Not that every area of fiber should have such, but
at least should they? Manhole covers "can" be keyed. For those of
you arguing that this is not enough, I would say at least it?s a start.


That is an option, but it doesn't address the real problem.

The real problem is route redundancy.  This is what the original contract
from DARPA to BBM, to create the Internet, was about!  "The net" was
created to enable communications bttn point A and point B in this exact
scenario.

No one should be surprised that ATT would cut-corners on critical
infrastructure. The good news is that this incident will likely result in
increased Federal scrutiny if not regulation.  We know how spectacularly
energy and banking deregulation failed.  Is that mistake being repeated
with telecommunications?

The bad news is that some of the $16M/yr ATT spends lobbying Congress (for
things like fighting number portability and getting a free pass on illegal
domestic surveillance) will likely be redirected to ask for money to "fix"
the problem they created.  This assumes ATT is as badly managed, and the US
FCC and DHS are better managed, than has been the case for the last 8
years.  Time will tell.

For a good "man in the street" perspective of how the outage effected
things like a pharmacy's ability to fill subscriptions and a university
computer's ability to boot check out a couple of shows broadcast on KUSP
(Santa Cruz Public Radio) this morning:

  http://www.jivamedia.com/askdrdawn/askdrdawn.php

  http://geekspeak.org/

Roger Marquis

Re: Fiber cut in SF area

2009-04-11 Thread Jorge Amodio

> The real problem is route redundancy.  This is what the original contract
> from DARPA to BBM, to create the Internet, was about!

s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/.

BBN won the contract to build the first four IMPs.

Theory and research about it is older, look at:
http://www.lk.cs.ucla.edu/LK/Bib/REPORT/PhD/proposal-01.html

But you are right, redundancy is the issue, cost is the factor.

Jorge.

RE: Fiber cut in SF area

2009-04-11 Thread Sean Donelan


On Sat, 11 Apr 2009, Roger Marquis wrote:

The real problem is route redundancy.  This is what the original contract
from DARPA to BBM, to create the Internet, was about!  "The net" was
created to enable communications bttn point A and point B in this exact
scenario.


Uh, not exactly.  There was diversity in this case, but there was also 
N+1 breaks.  Outside of a few counties in the Bay Area, the rest of the 
country's telecommunication system was unaffected.  So in that sense the 
system worked as designed.


Read the original DARPA papers, they were not about making sure grandma 
could still make a phone call.




For a good "man in the street" perspective of how the outage effected
things like a pharmacy's ability to fill subscriptions and a university
computer's ability to boot check out a couple of shows broadcast on KUSP
(Santa Cruz Public Radio) this morning:


Why didn't the "man in the street" pharmacy have its own backup plans?

Why didn't the pharmacy also have a COMCAST or RCN broadband connection 
for alternative Internet access besides AT&T or Verizon, a Citizens Band 
radio channel 9 for alternative emergency communications besides 9-1-1,
a satellite phone for alternative communications besides local cell 
phones, and a Hughes VSAT dish for yet even more diversity?  Why was the 
pharmacy relying on a single provider?  Or do it the old-fashion way 
before computers and telecommunications; keep a backup paper file of 
their records so they could continue to fill prescriptions?


Why didn't the pharmacy have more self-diversity? Probably the usual 
reason, more diversity costs more.  That may be the reason why hospitals 
have more diversity than neighborhood pharmacies; and emergency rooms 
have other ways to get medicine.  Maintaining diversity and backups is 
probably also part of the reason why filling a prescription at a hospital 
is much more expensive than filling a prescription at your neighborhood 
pharmacy.


Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait 
until the last minute to refill a critical presciption, have backup copies 
of prescriptions with her doctor, have an account with an alternative 
pharmacist in case her primary pharmacist isn't reachable, etc.


Readiness works better if everyone does their part, including grandma.

Next time it won't be AT&T, it will be Cox or Comcast or Qwest or Level 3 
or Global Crossing or  or  or  .  It won't be vandalism, it 
will be an earthquake, backhoe, gas main explosion, operator error, 


Everything fails sometimes.  What's your plan?

http://www.ready.gov/

personal opinion only

Re: Fiber cut in SF area

2009-04-11 Thread Mike Lyon

Anyone know how banks in the Bay Area did through this? I wonder how many
banks went dark and whether they had any backup plans/connectivity. Me
thinks its doubtful.

I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid,
Etc had thought about these kinds of issues? I personally doubt it. I bet
you they went dark along with everyone else. Unfortunate.

The funny thing is that the California lottery would be somewhat immuned to
this kind of disaster as they actually use Hughes VSAT at every single
retailer.

Sorry for the random thoughts...

-Mike


On Sat, Apr 11, 2009 at 4:11 PM, Sean Donelan  wrote:

> On Sat, 11 Apr 2009, Roger Marquis wrote:
>
>> The real problem is route redundancy.  This is what the original contract
>> from DARPA to BBM, to create the Internet, was about!  "The net" was
>> created to enable communications bttn point A and point B in this exact
>> scenario.
>>
>
> Uh, not exactly.  There was diversity in this case, but there was also N+1
> breaks.  Outside of a few counties in the Bay Area, the rest of the
> country's telecommunication system was unaffected.  So in that sense the
> system worked as designed.
>
> Read the original DARPA papers, they were not about making sure grandma
> could still make a phone call.
>
>
>  For a good "man in the street" perspective of how the outage effected
>> things like a pharmacy's ability to fill subscriptions and a university
>> computer's ability to boot check out a couple of shows broadcast on KUSP
>> (Santa Cruz Public Radio) this morning:
>>
>
> Why didn't the "man in the street" pharmacy have its own backup plans?
>
> Why didn't the pharmacy also have a COMCAST or RCN broadband connection for
> alternative Internet access besides AT&T or Verizon, a Citizens Band radio
> channel 9 for alternative emergency communications besides 9-1-1,
> a satellite phone for alternative communications besides local cell phones,
> and a Hughes VSAT dish for yet even more diversity?  Why was the pharmacy
> relying on a single provider?  Or do it the old-fashion way before computers
> and telecommunications; keep a backup paper file of their records so they
> could continue to fill prescriptions?
>
> Why didn't the pharmacy have more self-diversity? Probably the usual
> reason, more diversity costs more.  That may be the reason why hospitals
> have more diversity than neighborhood pharmacies; and emergency rooms have
> other ways to get medicine.  Maintaining diversity and backups is probably
> also part of the reason why filling a prescription at a hospital is much
> more expensive than filling a prescription at your neighborhood pharmacy.
>
> Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait
> until the last minute to refill a critical presciption, have backup copies
> of prescriptions with her doctor, have an account with an alternative
> pharmacist in case her primary pharmacist isn't reachable, etc.
>
> Readiness works better if everyone does their part, including grandma.
>
> Next time it won't be AT&T, it will be Cox or Comcast or Qwest or Level 3
> or Global Crossing or  or  or  .  It won't be vandalism, it will
> be an earthquake, backhoe, gas main explosion, operator error, 
>
> Everything fails sometimes.  What's your plan?
>
> http://www.ready.gov/
>
> personal opinion only
>
>

Re: Fiber cut in SF area

2009-04-11 Thread Jorge Amodio

> Read the original DARPA papers, they were not about making sure grandma
> could still make a phone call.

That's a great explanation in few words.

> Everything fails sometimes.  What's your plan?

Even the failover plans ...

Cheers
Jorge

Re: Fiber cut in SF area

2009-04-11 Thread Ravi Pina

While OT the news reports indicated ATMs were offline and many credit card
processing machines were down.  This is no big shock because many ATM
networks are on frame relay and POS credit card machines use POTS lines.

The outage also impacted mobile service too if it hadn't been said.

I hope we can put this thread to rest soon.

-r

On Sat, Apr 11, 2009 at 04:25:26PM -0700, Mike Lyon wrote:
> Anyone know how banks in the Bay Area did through this? I wonder how many
> banks went dark and whether they had any backup plans/connectivity. Me
> thinks its doubtful.
> 
> I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid,
> Etc had thought about these kinds of issues? I personally doubt it. I bet
> you they went dark along with everyone else. Unfortunate.
> 
> The funny thing is that the California lottery would be somewhat immuned to
> this kind of disaster as they actually use Hughes VSAT at every single
> retailer.
> 
> Sorry for the random thoughts...
> 
> -Mike
>

Re: Fiber cut in SF area

2009-04-11 Thread Roy

Mike Lyon wrote:
> Anyone know how banks in the Bay Area did through this? I wonder how many
> banks went dark and whether they had any backup plans/connectivity. Me
> thinks its doubtful.
>
> ...

Because of the loss of the alarm systems, many banks went to a method
where only one or two people were let in at a time.  Extra security was
also posted because of the inability to call 911.

Re: Fiber cut in SF area

2009-04-11 Thread Mike Lyon

Don't really care so much about the bank's security, especially if it was
one that received some the bailout money :)

I was more worried about if people could make withdraws from their bank
accounts. Deposits they could do as they could enter them in later but
withdraws I think would be different.

On Sat, Apr 11, 2009 at 5:19 PM, Roy  wrote:

> Mike Lyon wrote:
> > Anyone know how banks in the Bay Area did through this? I wonder how many
> > banks went dark and whether they had any backup plans/connectivity. Me
> > thinks its doubtful.
> >
> > ...
>
> Because of the loss of the alarm systems, many banks went to a method
> where only one or two people were let in at a time.  Extra security was
> also posted because of the inability to call 911.
>
>
>
>

Re: Fiber cut in SF area

2009-04-11 Thread Roy

Sean Donelan wrote:
> 
> Uh, not exactly.  There was diversity in this case, but there was also
> N+1 breaks.  Outside of a few counties in the Bay Area, the rest of
> the country's telecommunication system was unaffected.  So in that
> sense the system worked as designed.
> 

About eight or ten years ago I went to PacBell (or whatever it was
called at the time) and requested that two large facilities get a sonet
ring between them.  I was told I couldn't have it because they were both
fed through a single set of conduits and one backhoe could cut both
sides of the ring.  It wouldn't be diverse so they wouldn't provison it
unless I paid for the digging of new paths.

So much for their theory of diverse.  Sounds like the rules are
different for them.

There are one thing to also point out.  That train track next to the
manholes in South San Jose is the major line between the Bay Area and
Southern CA.  There are at least three or four fiber paths for different
companies buried along those tracks.  There are also connections from
Gilroy to the Hollister/San Juan Bautista area and thence to Salinas.  

It would have been very simple for the telcos to provision a backup path
southward.

RE: Fiber cut in SF area

2009-04-11 Thread Carlos Alcantar

I know as far as att/sbc/pacbell a lot of the time they run the ring
within the same conduit to at least have hardware protection on the
circuit I'm sure it's the same with other providers.

-carlos

-Original Message-
From: Roy [mailto:r.engehau...@gmail.com] 
Sent: Saturday, April 11, 2009 6:02 PM
To: nanog@nanog.org
Subject: Re: Fiber cut in SF area

Sean Donelan wrote:
> 
> Uh, not exactly.  There was diversity in this case, but there was also
> N+1 breaks.  Outside of a few counties in the Bay Area, the rest of
> the country's telecommunication system was unaffected.  So in that
> sense the system worked as designed.
> 

About eight or ten years ago I went to PacBell (or whatever it was
called at the time) and requested that two large facilities get a sonet
ring between them.  I was told I couldn't have it because they were both
fed through a single set of conduits and one backhoe could cut both
sides of the ring.  It wouldn't be diverse so they wouldn't provison it
unless I paid for the digging of new paths.

So much for their theory of diverse.  Sounds like the rules are
different for them.

There are one thing to also point out.  That train track next to the
manholes in South San Jose is the major line between the Bay Area and
Southern CA.  There are at least three or four fiber paths for different
companies buried along those tracks.  There are also connections from
Gilroy to the Hollister/San Juan Bautista area and thence to Salinas.  

It would have been very simple for the telcos to provision a backup path
southward.

Re: Fiber cut in SF area

2009-04-11 Thread Roger Marquis


Jorge Amodio wrote:

s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/.


/DARPA/ARPA/ may be splitting hairs.  According to

  http://www.livinginternet.com/i/ii_roberts.htm

"DARPA head Charlie Hertzfeld promised IPTO Director Bob Taylor a million
dollars to build a distributed communications network".

And apologies WRT /BBM/BBN/.  Guess it was really has been a while now
(given the 4 and 5 figure checks to BBN I signed back in the day).

Sean Donelan wrote:

On Sat, 11 Apr 2009, Roger Marquis wrote:

The real problem is route redundancy.  This is what the original contract
from DARPA to BBM, to create the Internet, was about!  "The net" was
created to enable communications bttn point A and point B in this exact
scenario.


Uh, not exactly.  There was diversity in this case, but there was also
N+1 breaks.  Outside of a few counties in the Bay Area, the rest of the
country's telecommunication system was unaffected.  So in that sense the
system worked as designed.

Read the original DARPA papers, they were not about making sure grandma
could still make a phone call.


Apparently even some network operators don't yet grasp the significance of
this event.


Why didn't the "man in the street" pharmacy have its own backup plans?


I assume they, as most of us, believed the government was taking care of
the country's critical infrastructure.  Interesting how well this
illustrates the growing importance of the Internet vis-a-vis other
communications channels.

Roger Marquis

Re: Fiber cut in SF area

2009-04-11 Thread Shane Ronan

An easy way to describe what your saying is "Security by obscurity is  
not security"


On Apr 11, 2009, at 8:31 AM, Joe Greco wrote:


JoÂ¢ wrote:

I'm confussed, but please pardon the ignorance.
All the data centers we have are at minimum keys to access
data areas. Not that every area of fiber should have such, but
at least should they? Manhole covers "can" be keyed. For those of
you arguing that this is not enough, I would say at least itâ€™s a  
start.

Yes if enough time goes by anything can happen, but how can one
argue an ATM machince that has (at times) thousands of dollars  
stands

out 24/7 without more immediate wealth. Perhaps I am missing
something here, do the Cops stake out those areas? dunno


The nice thing about the outdoors is how much of it there is.


Cute, but a lot of people seem to be wondering this, so a better  
answer

is deserved.

The ATM machine is somewhat protected for the extremely obvious reason
that it has cash in it, but an ATM is hardly impervious.

http://www.youtube.com/watch?v=4P8WM8ZZDHk

There are all sorts of strategies for attacking ATM's, and being
susceptible to a sledgehammer, crowbar, or truck smashing into the
unit shouldn't be hard to understand.

Most data centers have security that is designed to keep honest people
out of places that they shouldn't be.  Think that "security guard" at
the front will stop someone from running off with something valuable?
Maybe.  Have you considered following the emergency fire exits  
instead?

Running out the loading dock?  Etc?

Physical security is extremely difficult, and defending against a
determined, knowledgeable, and appropriately resourced attacker out to
get *you* is a losing battle, every time.

Think about a door.  You can close your bathroom door and set the  
privacy
lock, but any adult with a solid shoulder can break that door, or  
with a
pin (or flathead or whatever your particular knob uses) can stick it  
in
and trigger the unlock.  Your front door is more solid, but if it's  
wood,
and not reinforced, I'll give my steel-toed boots better than even  
odds

against it.  What?  You have a commercial hollow steel door?  Ok, that
beats all of that, let me go get my big crowbar, a little bending will
let me win.  Something more solid?  Ram it with a truck.  You got a
freakin' bank vault door?  Explosives, torches, etc.  Fort Knox?   
Bring a

large enough army, you'll still get in.

Notice a pattern?  For any given level of protection,  
countermeasures are
available.  Your house is best "secured" by making changes that make  
it
appear ordinary and non-attractive.  That means that a burglar is  
going to
look at your house, say "nah," and move on to your neighbor's house,  
where

your neighbor left the garage open.

But if I were a burglar and I really wanted in your house?  There's  
not
that much you could really do to stop me.  It's just a matter of how  
well

prepared I am, how well I plan.

So.  Now.  Fiber.

Here's the thing, now.  First off, there usually isn't a financial
motivation to attack fiber optic infrastructure.  ATM's get some
protection because without locks, criminals would just open them and
take the cash.  Having locks doesn't stop that, it just makes it  
harder.

However, the financial incentive for attacking a fiber line is low.
Glass is cheap.  We see attacks against copper because copper is
valuable, and yet we cannot realistically guard the zillions of miles
of copper that is all around.

Next.  Repair crews need to be able to access the manholes.  This is a
multifaceted problem.  First off, since there are so many manholes to
protect, and there are so many crews who might potentially need to  
access

them, you're probably stuck with a "standardized key" approach if you
want to lock them.  While this offers some protection against the  
average

person gaining unauthorized access, it does nothing to prevent "inside
job" attacks (and I'll note that this looks suspiciously like an  
"inside

job" of some sort).  Further, any locking mechanism can make it more
difficult to gain access when you really need access; some manholes  
are

not opened for years or even decades at a time.  What happens when the
locks are rusted shut?  Is the mechanism weak enough that it can be
forced open, or is it tolerable to have to wait extra hours while a
crew finds a way to open it?  Speaking of that, a manhole cover is
typically protecting some hole, accessway, or vault that's made out of
concrete.  Are you going to protect the concrete too?  If not, what
prevents me from simply breaking away the concrete around the manhole
cover rim (admittedly a lot of work) and just discarding the whole
thing?

Wait.  I just want to *break* the cable?  Screw all that.  Get me a
backhoe.  I'll just eyeball the direction I think the cable's going,
and start digging until I snag something.

Start to see the problems?

I'm not saying that security is a bad thing, just a tricky thing.

... JG
--
Joe Greco - sol.net N

Re: Fiber cut in SF area

2009-04-11 Thread Joel Jaeggli

Roger Marquis wrote:

>> Why didn't the "man in the street" pharmacy have its own backup plans?
> 
> I assume they, as most of us, believed the government was taking care of
> the country's critical infrastructure.  Interesting how well this
> illustrates the growing importance of the Internet vis-a-vis other
> communications channels.

It's also possible that they just planned on being down in such an event.

There's two factors here:

Not all low frequency risks are worth mitigating (how many of us have
generators at home).

Humans are bad at planning around rare events. Econimist Nassim Taleb's
book The Black Swan (isbn 978-1400063512) ought to be on everyones list
for coverage of the subject matter.

Fiber cuts are well outside the realm of experience for your average
business manager. The normal remediation strategy (for
telecommunications outage) in fact worked just fine, call your provider,
and or wait for them to fix it.

> Roger Marquis
>

Re: Fiber cut in SF area

2009-04-11 Thread Joe Greco

> An easy way to describe what your saying is "Security by obscurity is  
> not security"

Yes and no.  From a certain point of view, security is almost always 
closely tied to obscurity.

A cylinder lock is simply a device that operates through principles that
are relatively unknown to the average person:  they just know that you
stick a key in, turn it, and it opens.  The security of such a lock is
dependent on an attacker not knowing what a pin and tumbler design is, 
and not having the tools and (trivial) skills needed to defeat it.  That
is obscurity of one sort.

Public key crypto is, pretty much by definition, reliant on the obscurity
of private keys in order to make it work.

Ouch, eh.  And "hard to obtain" is essentially a parallel as well.
Simply making keyblanks hard to obtain is really a form of obscurity.
How much security is dependent on that sort of strategy?  It can (and
does) work well in many cases, but knowing the risks and limits is
important.

But that's all assuming that you're trying to secure something against
a typical attacker.

My point was more the inverse, which is that a determined, equipped,
and knowledgeable attacker is a very difficult thing to defend against.

Which brings me to a new point:  if we accept that "security by obscurity
is not security," then, what (practical thing) IS security?

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Fiber cut in SF area

2009-04-11 Thread Mike Lewinski


Joe Greco wrote:


My point was more the inverse, which is that a determined, equipped,
and knowledgeable attacker is a very difficult thing to defend against.


"The Untold Story of the World's Biggest Diamond Heist" published 
recently in Wired was a good read on that subject:


http://www.wired.com/politics/law/magazine/17-04/ff_diamonds


Which brings me to a new point:  if we accept that "security by obscurity
is not security," then, what (practical thing) IS security?


Obscurity as a principle works just fine provided the given token is 
obscure enough. Ideally there are layers of "security by obscurity" so 
compromise of any one token isn't enough by itself: my strong ssh 
password (1 layer of obscurity) is protected by the ssh server key (2nd 
layer) that is only accessible via vpn which has it's own encryption key 
(3rd layer). The loss of my password alone doesn't get anyone anything. 
The compromise of either the VPN or server ssh key (without already 
having direct access to those systems) doesn't get them my password either.


I think the problem is that the notion of "security by obscurity isn't 
security" was originally meant to convey to software vendors "don't rely 
on closed source to hide your bugs" and has since been mistakenly 
applied beyond that narrow context. In most of our applications, some 
form of obscurity is all we really have.


Mike

Re: Fiber cut in SF area

2009-04-11 Thread Mikael Abrahamsson


On Sat, 11 Apr 2009, Joe Greco wrote:

Public key crypto is, pretty much by definition, reliant on the 
obscurity of private keys in order to make it work.


In security terms, public key crypto is not "security by obscurity", as 
the obscurity part is related to how the method works, and the key is 
secret. So "openssh" is definitely not "security by obscurity", as anyone 
with programming knowledge can find out exactly how everything works, and 
the only thing that is a secret is the private key generated.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Fiber cut in SF area

2009-04-12 Thread Peter Beckman


On Sat, 11 Apr 2009, Christopher Morrow wrote:


I'm not sure that the manholes == atm discussion is valid, but in the
end the same thing is prone to happen to the manholes, there isn't
going to be a unique key per manhole, at best it'll be 1/region or
1/manhole-owner. In the end that key is compromised as soon as the
decision is made :(  Also keep in mind that keyed locks don't really
provide much protection, since anyone can order lockpicks over the
interwebs these days, even to states where ownership is apparently
illegal :(


 Too bad there isn't 1Password for manhole covers.

---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

Re: Fiber cut in SF area

2009-04-12 Thread Joe Greco

> 
> Joe Greco wrote:
> 
> > My point was more the inverse, which is that a determined, equipped,
> > and knowledgeable attacker is a very difficult thing to defend against.
> 
> "The Untold Story of the World's Biggest Diamond Heist" published 
> recently in Wired was a good read on that subject:
> 
> http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

> > Which brings me to a new point:  if we accept that "security by obscurity
> > is not security," then, what (practical thing) IS security?
> 
> Obscurity as a principle works just fine provided the given token is 
> obscure enough. 

Of course, but I said "if we accept that".  It was a challenge for the
previous poster.  ;-)

> Ideally there are layers of "security by obscurity" so 
> compromise of any one token isn't enough by itself: my strong ssh 
> password (1 layer of obscurity) is protected by the ssh server key (2nd 
> layer) that is only accessible via vpn which has it's own encryption key 
> (3rd layer). The loss of my password alone doesn't get anyone anything. 
> The compromise of either the VPN or server ssh key (without already 
> having direct access to those systems) doesn't get them my password either.
> 
> I think the problem is that the notion of "security by obscurity isn't 
> security" was originally meant to convey to software vendors "don't rely 
> on closed source to hide your bugs" and has since been mistakenly 
> applied beyond that narrow context. In most of our applications, some 
> form of obscurity is all we really have.

That's really it, and bringing us back to the fiber discussion, we are
forced, generally, to rely on obscurity.  In general, talk to a hundred
people on the street, few of them are likely to be able to tell you how
fiber gets from one city to another, or that a single fiber may be 
carrying immense amounts of traffic.  Most people expect that it just
all works somehow.  The fact that it's buried means that it is
sufficiently inaccessible to most people.  It will still be vulnerable
to certain risks, including backhoes, anything else that disrupts the
ground (freight derailments, earthquakes, etc), but those are all more
or less natural hazards that you protect against with redundancy.  The
guy who has technical specifics about your fiber network, and who picks
your vulnerable points and hits you with a hacksaw, that's just always
going to be much more complex to defend against.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Fiber cut in SF area

2009-04-13 Thread Stephen Sprunk


Mike Lewinski wrote:

Joe Greco wrote:
Which brings me to a new point:  if we accept that "security by 
obscurity is not security," then, what (practical thing) IS security?


Obscurity as a principle works just fine provided the given token is 
obscure enough. Ideally there are layers of "security by obscurity" so 
compromise of any one token isn't enough by itself: my strong ssh 
password (1 layer of obscurity) is protected by the ssh server key 
(2nd layer) that is only accessible via vpn which has it's own 
encryption key (3rd layer). The loss of my password alone doesn't get 
anyone anything. The compromise of either the VPN or server ssh key 
(without already having direct access to those systems) doesn't get 
them my password either.


I think the problem is that the notion of "security by obscurity isn't 
security" was originally meant to convey to software vendors "don't 
rely on closed source to hide your bugs" and has since been mistakenly 
applied beyond that narrow context. In most of our applications, some 
form of obscurity is all we really have.


The accepted standard is that a system is secure iff you can disclose 
_all_ of the details of how the system works to an attacker _except_ the 
private key and they still cannot get in -- and that is true of most 
open-standard or open-source encryption/security products due to 
extensive peer review and iterative improvements.  What "security by 
obscurity" refers to are systems so weak that their workings cannot be 
exposed because then the keys will not be needed, which is true of most 
closed-source systems.  It does _not_ refer to keeping your private keys 
secret.


Key management is considered to be an entirely different problem.  If 
you do not keep your private keys secure, no security system will be 
able to help you.


S

--
Stephen Sprunk "God does not play dice."  --Albert Einstein
CCIE #3723 "God is an inveterate gambler, and He throws the
K5SSSdice at every possible opportunity." --Stephen Hawking



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Fiber cut in SF area

2009-04-13 Thread Steven M. Bellovin

On Mon, 13 Apr 2009 09:18:04 -0500
Stephen Sprunk  wrote:

> Mike Lewinski wrote:
> > Joe Greco wrote:
> >> Which brings me to a new point:  if we accept that "security by 
> >> obscurity is not security," then, what (practical thing) IS
> >> security?
> >
> > Obscurity as a principle works just fine provided the given token
> > is obscure enough. Ideally there are layers of "security by
> > obscurity" so compromise of any one token isn't enough by itself:
> > my strong ssh password (1 layer of obscurity) is protected by the
> > ssh server key (2nd layer) that is only accessible via vpn which
> > has it's own encryption key (3rd layer). The loss of my password
> > alone doesn't get anyone anything. The compromise of either the VPN
> > or server ssh key (without already having direct access to those
> > systems) doesn't get them my password either.
> >
> > I think the problem is that the notion of "security by obscurity
> > isn't security" was originally meant to convey to software vendors
> > "don't rely on closed source to hide your bugs" and has since been
> > mistakenly applied beyond that narrow context. In most of our
> > applications, some form of obscurity is all we really have.
> 
> The accepted standard is that a system is secure iff you can disclose 
> _all_ of the details of how the system works to an attacker _except_
> the private key and they still cannot get in -- and that is true of
> most open-standard or open-source encryption/security products due to 
> extensive peer review and iterative improvements.  What "security by 
> obscurity" refers to are systems so weak that their workings cannot
> be exposed because then the keys will not be needed, which is true of
> most closed-source systems.  It does _not_ refer to keeping your
> private keys secret.

Correct.  Open source and open standards are (some) ways to achieve that
goal. They're not the only ones, nor are they sufficient.  (Consider
WEP as a glaring example of a failure of a standards process.)  On the
other hand, I was once told by someone from NSA that they design all of
their gear on the assumption that Serial #1 of any new crypto device is
delivered to the Kremlin.

This principle, as applied to cryptography, was set out by Kerckhoffs
in 1883; see http://www.petitcolas.net/fabien/kerckhoffs/ for details.
> 
> Key management is considered to be an entirely different problem.  If 
> you do not keep your private keys secure, no security system will be 
> able to help you.
> 
Yes.  One friend of mine likens insecurity to entropy: you can't
destroy it, but you can move it around.  For example, cryptography lets
you trade the insecurity of the link for the insecurity of the key, on
the assumption that you can more easily protect a few keys than many
kilometers of wire/fiber/radio.


--Steve Bellovin, http://www.cs.columbia.edu/~smb

RE: Fiber cut in SF area

2009-04-13 Thread Dylan Ebner

One thing that is missing here is before we can define "security" we
need to define the "threat" and the "obstruction" the security creates.
With an ATM machine, the threat is someone comes and steals the machine
for the cash. The majority of the assailants in an ATM case are not
interested in the access passwords, so that is not viewed as a threat by
the bank. Then bank then says, "If we set really complicated passwords,
our repair guys (or contractors) will not be able to fix them." So
setting hard passwords is an obstruction. This happens every day, in
every IT department in the world. 

So lets define the "Threat" to the fiber network? We know it isn't
monetary as their isn't much value in selling cut sections of fiber. So
that leaves out your typical ATM theif. That leaves us with directed
attack, revenge or pure vandalism.

In a directed attack or revenge scenario, which is what this case looks
like, how are manhole locks going to help? If it is was the fiber union,
wouldn't they already have the keys anyway? If this was some kind of
terrorism scenario wouldn't they also have the resources to get the
keys, either by getting employed by the phone company or the fiber union
or any one of the other thousand companies that would need those keys?

Manhole locks are just going to stop vandalism, and I think the threat
to obstruction calculation just doesn't add up for that small level of
isolated cases.

Here in Qwest territory, manhole locks would be disasterours for repair
times. We have had times when our MOE network has an outage and Qwest
cannot fix the problem because their repair guys don't have the keys to
their own buildings. Seriously. Their own buildings.

Ultimately, what really needs to be addresses is the redundancy problem.
And this needs to be addresses by everyone who was affected, not just
ATT and Verizon, etc. 

A few years ago we had a site go down when a sprint DS-3 was cut. This
was a major wake-up call for us because we had 2 t-1's for the site and
they were suppose to have path divergence. And they did, up to the qwest
CO where they handed off the circuit to sprint. In the end, we built in
workflow redundancies so if any site goes down, we can still operate at
near 100% capacity. 

My point is, it is getting harder and harder to gurantee path divergence
and sometimes the redundancies need to be built into the workflow
instead of IT. 

But that does't mean we cannot try. I remember during Katrima a
datacenter in downtown New Orleans managed to stay online for the
duration of disaster. These guys were on the ball and it paid off for
them. 

In the end, as much as I like to blame the phone companies when we have
problems, I also have to take some level of responsibility. And with
each of these types of incidents we learn. For everyone affected, you
now know even though you have two carriers, you do not have path
divergence. And for everyone who colos at an affected Datacenter and
get's your service from that center, you know they don't have
divergence. So we need to ask ourselves, "where do we go from here?"

It will be easier to get more divergence than secure all the manholes in
the country. 

Dylan Ebner, Network Engineer
Consulting Radiologists, Ltd.
1221 Nicollet Mall, Minneapolis, MN 55403
ph. 612.573.2236 fax. 612.573.2250
dylan.eb...@crlmed.com
www.consultingradiologists.com

-Original Message-
From: Joe Greco [mailto:jgr...@ns.sol.net] 
Sent: Sunday, April 12, 2009 7:12 AM
To: Mike Lewinski
Cc: nanog@nanog.org
Subject: Re: Fiber cut in SF area

> 
> Joe Greco wrote:
> 
> > My point was more the inverse, which is that a determined, equipped,

> > and knowledgeable attacker is a very difficult thing to defend
against.
> 
> "The Untold Story of the World's Biggest Diamond Heist" published 
> recently in Wired was a good read on that subject:
> 
> http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

> > Which brings me to a new point:  if we accept that "security by 
> > obscurity is not security," then, what (practical thing) IS
security?
> 
> Obscurity as a principle works just fine provided the given token is 
> obscure enough.

Of course, but I said "if we accept that".  It was a challenge for the
previous poster.  ;-)

> Ideally there are layers of "security by obscurity" so compromise of 
> any one token isn't enough by itself: my strong ssh password (1 layer 
> of obscurity) is protected by the ssh server key (2nd
> layer) that is only accessible via vpn which has it's own encryption 
> key (3rd layer). The loss of my password alone doesn't get anyone
anything.
> The compromise of either the VPN or server ssh key (without already 
> having direct access to those systems

RE: Fiber cut in SF area

2009-04-13 Thread Mikael Abrahamsson


On Mon, 13 Apr 2009, Dylan Ebner wrote:


Manhole locks are just going to stop vandalism, and I think the threat
to obstruction calculation just doesn't add up for that small level of
isolated cases.


It doesn't stop it, it just makes it slightly harder, and they'll go after 
another point.




This is the bay area as well... How long do you need to spend with a torch 
to cut thru that? A couple of minutes?


There is absolutely no way you can stop a determined attacker, and it 
would increase cost a lot more than it's worth. Time is better spent 
stopping the few people who actually do these kinds of things, same way as 
it's not worth it for regular people to wear body armour all the time, 
just in case they might get shot, or have parachutes and emergency exits 
that work in mid-flight on commercial airliners. The various police 
agencies and the NTSB cost less in a cost/benefit analysis.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Fiber cut in SF area

2009-04-13 Thread joel . mercado

It all comes down to money... It will cost them lots of it to get power and 
some type of readers installed to monitor manhole access... There has always 
been a lack of security on the telco side, this incident just brings it to 
light... In my town many of the verizon fios boxes are not locked and the 
wiring frame boxes for pots line neither.. Its all of a matter of how much cash 
they wanna throw at it...
Sent on the Now Network� from my Sprint® BlackBerry

-Original Message-
From: "Dylan Ebner" 

Date: Mon, 13 Apr 2009 09:57:30 
To: 
Subject: RE: Fiber cut in SF area

One thing that is missing here is before we can define "security" we
need to define the "threat" and the "obstruction" the security creates.
With an ATM machine, the threat is someone comes and steals the machine
for the cash. The majority of the assailants in an ATM case are not
interested in the access passwords, so that is not viewed as a threat by
the bank. Then bank then says, "If we set really complicated passwords,
our repair guys (or contractors) will not be able to fix them." So
setting hard passwords is an obstruction. This happens every day, in
every IT department in the world. 

So lets define the "Threat" to the fiber network? We know it isn't
monetary as their isn't much value in selling cut sections of fiber. So
that leaves out your typical ATM theif. That leaves us with directed
attack, revenge or pure vandalism.

In a directed attack or revenge scenario, which is what this case looks
like, how are manhole locks going to help? If it is was the fiber union,
wouldn't they already have the keys anyway? If this was some kind of
terrorism scenario wouldn't they also have the resources to get the
keys, either by getting employed by the phone company or the fiber union
or any one of the other thousand companies that would need those keys?

Manhole locks are just going to stop vandalism, and I think the threat
to obstruction calculation just doesn't add up for that small level of
isolated cases.

Here in Qwest territory, manhole locks would be disasterours for repair
times. We have had times when our MOE network has an outage and Qwest
cannot fix the problem because their repair guys don't have the keys to
their own buildings. Seriously. Their own buildings.

Ultimately, what really needs to be addresses is the redundancy problem.
And this needs to be addresses by everyone who was affected, not just
ATT and Verizon, etc. 

A few years ago we had a site go down when a sprint DS-3 was cut. This
was a major wake-up call for us because we had 2 t-1's for the site and
they were suppose to have path divergence. And they did, up to the qwest
CO where they handed off the circuit to sprint. In the end, we built in
workflow redundancies so if any site goes down, we can still operate at
near 100% capacity. 

My point is, it is getting harder and harder to gurantee path divergence
and sometimes the redundancies need to be built into the workflow
instead of IT. 

But that does't mean we cannot try. I remember during Katrima a
datacenter in downtown New Orleans managed to stay online for the
duration of disaster. These guys were on the ball and it paid off for
them. 

In the end, as much as I like to blame the phone companies when we have
problems, I also have to take some level of responsibility. And with
each of these types of incidents we learn. For everyone affected, you
now know even though you have two carriers, you do not have path
divergence. And for everyone who colos at an affected Datacenter and
get's your service from that center, you know they don't have
divergence. So we need to ask ourselves, "where do we go from here?"

It will be easier to get more divergence than secure all the manholes in
the country. 

Dylan Ebner, Network Engineer
Consulting Radiologists, Ltd.
1221 Nicollet Mall, Minneapolis, MN 55403
ph. 612.573.2236 fax. 612.573.2250
dylan.eb...@crlmed.com
www.consultingradiologists.com

-Original Message-
From: Joe Greco [mailto:jgr...@ns.sol.net] 
Sent: Sunday, April 12, 2009 7:12 AM
To: Mike Lewinski
Cc: nanog@nanog.org
Subject: Re: Fiber cut in SF area

> 
> Joe Greco wrote:
> 
> > My point was more the inverse, which is that a determined, equipped,

> > and knowledgeable attacker is a very difficult thing to defend
against.
> 
> "The Untold Story of the World's Biggest Diamond Heist" published 
> recently in Wired was a good read on that subject:
> 
> http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

> > Which brings me to a new point:  if we accept that "security by 
> > obscurity is not security," then, what (practical thing) IS
security?
> 
> Obscurity as a principle works just fine provided the given token is 
> obscure enough.

Of course, but I

Re: Fiber cut in SF area

2009-04-13 Thread Andy Ringsmuth



On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote:

Manhole locks are just going to stop vandalism, and I think the  
threat
to obstruction calculation just doesn't add up for that small level  
of

isolated cases.


It doesn't stop it, it just makes it slightly harder, and they'll go  
after another point.


IMHO, I think manhole locks would only serve to HEIGHTEN the threat,  
not minimize it.  Flag this under the whole "obscurity" category, but  
think about this - if you're a vandal itching to do something stupid,  
and you see a bunch of manhole covers and a couple of them have locks  
on them, which ones are you going to target?  The ones with the locks,  
of course.  Why?  Because by the very existence of the locks, it  
implies there's something of considerable value beyond the lock.



-Andy

Re: Fiber cut in SF area

2009-04-13 Thread Matthew Petach

On 4/13/09, Dylan Ebner  wrote:
>  My point is, it is getting harder and harder to gurantee path divergence
>  and sometimes the redundancies need to be built into the workflow
>  instead of IT.

Actually, in many ways it's getting easier; now, you can sign an NDA
with your fiber providers and get GIS data for the fiber runs which you can
pop into Google Earth, and verify path separation along the entire run;
you put notification requirements into the contract stipulating that the
fiber provider *must* notify you and provide updated GIS data if the
path must be physically moved, and the move deviates the path by
more than 50 feet from the previous GIS data; and you put escape
clauses into the contract in case the re-routing of the fiber unavoidably
reduces or eliminates your physical run diversity from your other
providers.

In years past, trying to overlay physical map printouts to validate
path separation was a nightmare.  Now, standardized GIS data
formats make it a breeze.

"protected rings" are a technology of the past.  Don't count on your
vendor to provide "redundancy" for you.  Get two unprotected runs
for half the cost each, from two different providers, and verify the
path separation and diversity yourself with GIS data from the two
providers; handle the failover yourself.  That way, you *know* what
your risks and potential impact scenarios are.  It adds a bit of
initial planning overhead, but in the long run, it generally costs a
similar amount for two unprotected runs as it does to get a
protected run, and you can plan your survival scenarios *much*
better, including surviving things like one provider going under,
work stoppages at one provider, etc.

Sometimes a little bit of paranoia can help save your butt...or at
least keep you out of the hot seat.

Matt

Re: Fiber cut in SF area

2009-04-13 Thread Dorn Hetzel

I guess the next generation fiber networks will need to be installed with
tunnel boring machines and just not surface anywhere except the endpoints
:)  After all, undersea cables get along just fine without convenient access
along their length...

On Mon, Apr 13, 2009 at 12:12 PM, Mikael Abrahamsson wrote:

> On Mon, 13 Apr 2009, Dylan Ebner wrote:
>
>  Manhole locks are just going to stop vandalism, and I think the threat
>> to obstruction calculation just doesn't add up for that small level of
>> isolated cases.
>>
>
> It doesn't stop it, it just makes it slightly harder, and they'll go after
> another point.
>
> 
>
> This is the bay area as well... How long do you need to spend with a torch
> to cut thru that? A couple of minutes?
>
> There is absolutely no way you can stop a determined attacker, and it would
> increase cost a lot more than it's worth. Time is better spent stopping the
> few people who actually do these kinds of things, same way as it's not worth
> it for regular people to wear body armour all the time, just in case they
> might get shot, or have parachutes and emergency exits that work in
> mid-flight on commercial airliners. The various police agencies and the NTSB
> cost less in a cost/benefit analysis.
>
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
>

Re: Fiber cut in SF area

2009-04-13 Thread Dorn Hetzel

Or skip the locks and fill the manholes with sand.  Then provide the service
folks those big suction trucks to remove the sand for servicing :)

On Mon, Apr 13, 2009 at 12:28 PM, Andy Ringsmuth wrote:

>
> On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote:
>
>  Manhole locks are just going to stop vandalism, and I think the threat
>>> to obstruction calculation just doesn't add up for that small level of
>>> isolated cases.
>>>
>>
>> It doesn't stop it, it just makes it slightly harder, and they'll go after
>> another point.
>>
>
> IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not
> minimize it.  Flag this under the whole "obscurity" category, but think
> about this - if you're a vandal itching to do something stupid, and you see
> a bunch of manhole covers and a couple of them have locks on them, which
> ones are you going to target?  The ones with the locks, of course.  Why?
>  Because by the very existence of the locks, it implies there's something of
> considerable value beyond the lock.
>
>
> -Andy
>
>

Re: Fiber cut in SF area

2009-04-13 Thread Justin M. Streiner


On Mon, 13 Apr 2009, Dorn Hetzel wrote:


I guess the next generation fiber networks will need to be installed with
tunnel boring machines and just not surface anywhere except the endpoints
:) After all, undersea cables get along just fine without convenient 
access along their length...


Boat anchors and earthquakes do a pretty effective job of cutting 
submarine cables.


jms

RE: Fiber cut in SF area


On Mon, 13 Apr 2009, Dylan Ebner wrote:


It will be easier to get more divergence than secure all the manholes in
the country.


 I still think skipping the securing of manholes and access points in favor
 of active monitoring with offsite access is a better solution.  You can't
 keep people out, especially since these manholes and tunnels are designed
 FOR human access.  But a better job can be done of monitoring and knowing
 what is going on in the tunnels and access points from a remote location.

Cheap: light sensor + cell phone = knowing exactly when and where the
amount of light in the tunnel changes.  Detects unauthorized
intrusions.  Make sure to detect all visible and IR spectrum, should
someone very determined use night vision and IR lights to disable the
sensor.

Mid-Range: Webcam + cell phone = SEEING what is going on plus
everything above.

High-end: Webcam + cell phone + wifi or wimax backup both watching the
entrance and the tunnels.

James Bond: Lasers.

 Active monitoring of each site makes sure each one is online.

 Pros:
* Knowing immediately that there is a change in environment in your
  tunnels.
* Knowing who or at least THAT something is in there
* Being able to proactively mitigate attempts
* Availability of Arduino, SIM card adapters, and sophisticated sensor
  and camera equipment at low cost

 Cons:
* Cell provider outage or spectrum blocker removes live notifications
* False positives are problematic and can lower monitoring thresholds
* Initial expense of deployment of monitoring systems

 Farmers use tiny embedded devices on their farms to monitor moisture,
 rain, etc. in multiple locations to customize irrigation and to help avoid
 loss of crops.  These devices communicate with themselves, eventually
 getting back to a main listening post which relays the information to the
 farmer's computers.

 Tiny, embedded, networked devices that monitor the environment in the
 tunnels that run our fiber to help avoid loss of critical communications
 services seems to be a good idea.  Cheap, disposable devices that can
 communicate with each other as well as back to some HQ is a way to at
 least know about problems of access before they happen.  No keys to lose,
 no technology keeping people out and causing repair problems.

 Some other things that could detect access problems:
* Pressure sensors (maybe an open manhole causes a detectable change in
  air pressure in the tunnel)
* Temperature sensors (placed near access points, detects welding and
  thermite use)
* Audio monitor (can help determine if an alert is just a rat squealing
  or people talking -- could even be automated to detect certain types of
  noises)
* IR (heat) motion detection, as long as giant rats/rodents aren't a problem
* Humidity sensors (sell the data to weatherbug!)

 One last thought inspired by the guy who posted about pouring quick-set
 concrete in to slow repair.  Get some heavy-duty bags, about 10 feet long
 and large enough to fill the space in the tunnel.  More heavily secure the
 fiber runs directly around the access space, then inflate two bags on
 either side of the access point.  Easily deflated, these devices also have
 an electronic device which can notify HQ that they are being deflated or
 the pressure inside is changing (indicating pushing or manipulation).
 That way you only need to put these bags at access points, not throughout
 the whole tunnel.

 Kinda low-tech, but could be effective.  No keys needed, could be
 inflated/deflated quickly, and you still get notification back to a
 monitoring point.

Beckman
---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

RE: Fiber cut in SF area

On Mon, 13 Apr 2009, chris.ra...@nokia.com wrote:

Peter Beckman [mailto:beck...@angryox.com] wrote:

Sent: Monday, April 13, 2009 11:19 AM
To: Dylan Ebner
Cc: nanog@nanog.org
Subject: RE: Fiber cut in SF area

On Mon, 13 Apr 2009, Dylan Ebner wrote:

It will be easier to get more divergence than secure all the
manholes in the country.

I still think skipping the securing of manholes and access
points in favor of active monitoring with offsite access is a
better solution.

The only thing missing from your plan was a cost analysis.  Cost of each,
plus operational costs, * however many of each type.  How much would that
be?

 So, let's see.  I'm pulling numbers out of my butt here, but basing it on
 non-quantity-discounted hardware available off the shelf.

 $500,000 to get it built with off-the-shelf components, tested in hostile
 tunnel environments and functioning.

 Then $350 per device, which would cover 1000 feet of tunnel, or about
 $2000 per mile for the devices.  I'm not sure how things are powered in
 the tunnels, so power may need to be run, or the system could run off
 sealed-gel batteries (easily replaced and cheap, powers device for a
 year), system can be extremely low power.  Add a communication device
 ($1000) every mile or two (the devices communicate between themselves back
 to the nearest communications device).

 Total cost, assuming 3 year life span of the device, is about $3000 per
 mile for equipment, or $1000 per year for equipment, plus $500 per year
 per mile for maintenance (batteries, service contracts, etc).  Assumes
 your existing cost of tunnel maintenance can also either replace devices
 or batteries or both.

 Add a speedy roomba like RC device in the tunnel with an HD cam and a 10
 or 20 mile range between charging stations that can move to the location
 where an anomaly was detected, and save some money on the per-device cost.
 It could run on an overhead monorail, or just wheels, depending on the
 tunnel configuration and moisture content.

 Add yet another system -- an alarm of sorts -- that goes off upon any
 anomaly being detected, and goes off after 5 minutes of no detection, to
 thwart teenagers and people who don't know how sophisticated the
 monitoring system really is.  Put the alarm half way between access
 points, so it is difficult to get to and disable.

 Network it all, so that it can be controlled and updated from a certain
 set of IPs, make sure all changes are authenticated using PKI or
 certificates, and now you've made it harder to hack.  Bonus points -- get
 a communication device that posts updates via SSL to multiple
 pre-programmed or random Confickr-type domains to make sure the system
 continues to be able to communicate in the event of a large outage.

Then amortize that out to our bills.  Extra credit: would you pay for it?

 Assuming bills in the hundreds of thousands of dollars per month, maybe to
 the millions of dollars, and then figure out what an outage costs you
 according to the SLAs.

 Then figure out how much a breach and subsequent fiber cut costs you in
 SLA payouts or credits, multiply by 25%, and that's your budget.  If the
 proposed system is less, why wouldn't you do it?

 The idea is inspired by the way Google does their datacenters -- use
 cheap, off-the-shelf hardware, network it together in smart ways, make it
 energy efficient, ... profit!

 Anyone want to invest?  Maybe I should start the business.

Beckman
---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

RE: Fiber cut in SF area

2009-04-13 Thread Scott Weeks

--- beck...@angryox.com wrote:

>> I still think skipping the securing of manholes and access
>> points in favor of active monitoring with offsite access is a
>> better solution.
>
> The only thing missing from your plan was a cost analysis.  Cost of each,
> plus operational costs, * however many of each type.  How much would that
> be?

  So, let's see.  I'm pulling numbers out of my butt here, but basing it on
  non-quantity-discounted hardware available off the shelf.
-

Manpower to design, build, maintain, train folks and monitor in the NOC.  Costs 
of EMS, its maintenance.  blah, blah, blah...

scott

RE: Fiber cut in SF area


On Mon, 13 Apr 2009, Scott Weeks wrote:




--- beck...@angryox.com wrote:


I still think skipping the securing of manholes and access
points in favor of active monitoring with offsite access is a
better solution.


The only thing missing from your plan was a cost analysis.  Cost of each,
plus operational costs, * however many of each type.  How much would that
be?


 So, let's see.  I'm pulling numbers out of my butt here, but basing it on
 non-quantity-discounted hardware available off the shelf.
-


Manpower to design, build, maintain, train folks and monitor in the NOC.
Costs of EMS, its maintenance.  blah, blah, blah...


 My estimates are for getting something off the ground, equipment-wise, not
 operationally.

 What is the cost of the outages?  And if this setup can detect un-reported
 backhoe activity via accelerometers BEFORE it slices through the cable and
 you can get someone out to investigate the activity before it gets cut,
 how much is that worth?

 And my estimate was for the hardware, not training, etc.  I'm guessing
 existing NOCs can easily incorporate new SNMP traps or other methods of
 alerts into their system fairly easily.

Beckman
---
Peter Beckman  Internet Guy
beck...@angryox.com http://www.angryox.com/
---

RE: Fiber cut in SF area

2009-04-13 Thread chris.ranch

Peter Beckman [mailto:beck...@angryox.com] wrote:
>Sent: Monday, April 13, 2009 11:19 AM
>To: Dylan Ebner
>Cc: nanog@nanog.org
>Subject: RE: Fiber cut in SF area
>
>On Mon, 13 Apr 2009, Dylan Ebner wrote:
>
>> It will be easier to get more divergence than secure all the 
>> manholes in the country.
>
>I still think skipping the securing of manholes and access 
>points in favor of active monitoring with offsite access is a 
>better solution.  

The only thing missing from your plan was a cost analysis.  Cost of each, plus 
operational costs, * however many of each type.  How much would that be?

Then amortize that out to our bills.  Extra credit: would you pay for it?

Chris

RE: Fiber cut in SF area

2009-04-13 Thread Crist Clark

>>> On 4/13/2009 at 1:12 PM, Peter Beckman  wrote:
> On Mon, 13 Apr 2009, Scott Weeks wrote:
> 
>>
>>
>> --- beck...@angryox.com wrote:
>>
 I still think skipping the securing of manholes and access
 points in favor of active monitoring with offsite access is a
 better solution.
>>>
>>> The only thing missing from your plan was a cost analysis.  Cost of each,
>>> plus operational costs, * however many of each type.  How much would that
>>> be?
>>
>>  So, let's see.  I'm pulling numbers out of my butt here, but basing it on
>>  non-quantity-discounted hardware available off the shelf.
>> -
>>
>>
>> Manpower to design, build, maintain, train folks and monitor in the NOC.
>> Costs of EMS, its maintenance.  blah, blah, blah...
> 
>   My estimates are for getting something off the ground, equipment-wise, not
>   operationally.
> 
>   What is the cost of the outages?

But would alarms prevent any, or what proportion, of these incidents?
>From what we know of this specific one, would an alarm have stopped
the perpetrator(s)? It would have bought the NOC five, ten minutes
tops before they got the alarm on the circuit. And in practice would
a manhole alarm translate to a call to Homeland Security to have
the SEALs descend the site pronto, a police unit to roll by when it
has the time, or is it going to be an AT&T truck rolling by between
calls? I'm guessing number two or three, probably three. So what
would it get them in this case. If it doesn't deter these guys,
who does it deter?

And what are the costs of false alarms? What will the ratio of
"real" alarms to false ones be? Maybe lower-stakes vandals take to
popping the edge of manhole covers as a little prank. Or that one
that triggers whenever a truck tire hits it right. Or the whole line
of them that go off whenever the temperature drops below freezing.
Or, what I am absolutely sure will happen, miscommunication between
repair crews and the NOC about which ones are being moved or field
crews opening them without warning the NOC (or even intra-NOC
communication). Will they be a boy who cried wolf?

RE: Fiber cut in SF area

2009-04-13 Thread chris.ranch

Hi Peter,

You wrote:
>  So, let's see.  I'm pulling numbers out of my butt here, 

>  Total cost...is about $3000 per mile for equipment

> It could run on an overhead monorail

> Network it all

> Confickr-type domains to make sure 

I get the feeling you haven't deployed or operated large networks.  You never 
did say what the multiplier was.  How many miles or detection nodes there were. 
 Think millions.  The number that popped into my head when thinking of active 
detection measures for the physical network is $billions.

Joel is right: the thing about the outdoors is there's a lot of it.  The cost 
over time investment of copper and fiber communucations networks, power 
transmission networks, cable transmission networks is pretty well documented 
elsewhere.  Google around a little for them.  The investment is tremendous.

All for a couple of minutes advanced notice of an outage?  Would it reduce the 
risk?  No.  Would it reduce the MTBF or MTTR?  No.  Of all outages, how often 
does this scenario (or one that would trigger your alarm) occur?  I'm sure it's 
down on the list.

>> Then amortize that out to our bills.  Extra credit: would 
>you pay for it?
>
>  Assuming bills in the hundreds of thousands of dollars per 
>month, maybe to
>  the millions of dollars, and then figure out what an outage costs you
>  according to the SLAs.
>
>  Then figure out how much a breach and subsequent fiber cut 
>costs you in
>  SLA payouts or credits, multiply by 25%, and that's your 
>budget.  If the
>  proposed system is less, why wouldn't you do it?

SLA's account for force de majure (including sabotage), so I really doubt there 
will be any credits.  In fact, there will likely be an uptick on spending as 
those who really need nines build multi-provider multi-path diversity.  Here 
come the microwave towers!

>  The idea is inspired by the way Google does their datacenters -- use
>  cheap, off-the-shelf hardware, network it together in smart 
>ways, make it
>  energy efficient, ... profit!

Works great inside four walls. 

>  Anyone want to invest?  Maybe I should start the business.

Nahh, I already have a web cam on my Smarties orb.  What else do I really need?

Chris

RE: Fiber cut in SF area