RE: Postmaster @ vtext.com (or what are best practice to send SMS these days)

[Randy Epstein]

David Ulevitch wrote:



> What else are operators doing to get the pages out when things go wonky?

Get a pager!  :)  SMS is just not as reliable.

> David

Randy

RE: YouTube IP Hijacking

[Randy Epstein]

Arnd wrote:

> You _need_ a license to drive a car, fly a plane etc. but until now you
> dont need to show that youre skilled enough to run a border router. Good
> idea? I dont think so.

My point was that even with a license, accidents still occur.

> I believe that people who run ASNs should have the knowledge for it and
> that _someone_ should test this. Right now the LIRs seems to be the best
> institution for this. And no, i dont think the vendors should do this.

Vendors currently do train their customers and certify them.  LIRs don't and
cannot know all the gear out there and configurations from network to
network vary.  This doesn't stop route leaks, nor would this protect us from
intentional mischief.  I'm not saying it can't happen, but most leaks are
caused by accident, and I might add by trained personnel and untrained
personnel alike.

Many of the suggestions that we've been seeing regarding this subject have
pros and cons, but some even solve both problems: both accidental and
intentional leaks.

I am not against training personnel, but your solution doesn't resolve
either of the above for the most part.

> -- Arnd

Randy

RE: YouTube IP Hijacking

[Randy Epstein]

>>This isn't the answer.  If it were, there would be no car accidents, pilot
>>error caused plane crashes, etc.

> Probably the reason you dont need to have a pilot license...

Sorry, what?

> Dont get me wrong: I not the "Policy this/that" type but i think its a
> good idea to ensure that ppl who run "basic network infrastructure" have
> minimal clue of how to do this.

Do you really believe that LIRs should be administering tests before issuing
ASNs?  Should vendors do the same prior to selling their gear?  Take this
further, electric company should require its customers to take a test before
they are allowed to order service for fear they might electrocute themselves
or the water company fearing customers may drown?

> -- Arnd

Randy

RE: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]

[Randy Epstein]

Valdis wrote:

> He explicitly said "single-homed".  Of course, multi-homed requires
> different handling, because you may hear their other home announce them
> (although again, you probably shouldn't listen to *THAT* announcement
> either if *your* link to them is up).  And I posit that if you don't know 
> if your customer is single or multi-homed, you have *bigger* issues to
> deal with.

My bad, I misread his multi-homed comment.  From what I understand (and have
seen in practice) PCCW does not listen to their address space from their
peers no matter what the status of the connection to their customer is.  I
find this policy flat out flawed.

Randy

RE: BGP prefix filtering, how exactly? [Re: YouTube IP Hijacking]

[Randy Epstein]

> Our own or our singlehomed customers' address space -- we would reject 
> such an advertisement.  The same inbound consistency check applies to 
> peers and upstreams/transits.
>
> If it's someone else's or a more specific or the same prefix as our 
> multihomed customers -- we accept it.  There isn't anything else we 
> can do in practise which would not hurt legitimate routing..


What do you do when one of your multi-homed customers on your IP space has
an outage on their connection to your network?  How would your customers
then reach that customer? 

Although this wouldn't be THAT BIG of a deal for small networks, if say a
larger or a Tier-1 provider practiced this (AFAIK, the only somewhat large
network to do this is, believe it or not, PCCW), your customer would
experience a major outage.

There must be a better way.  :)

> Pekka Savola

Regards,

Randy

RE: YouTube IP Hijacking

[Randy Epstein]

Tomas L. Byrnes wrote:

> Perhaps certain ASes that are considered "high priority", like Google,
> YouTube, Yahoo, MS (at least their update servers), can be trusted to
> propagate routes that are not aggregated/filtered, so as to give them
> control over their reachability and immunity to longer-prefix hijacking
> (especially problematic with things like MS update sites).

Not to stir up a huge debate here, but if I were a day trader, I could live
without YouTube for a day, but not e*trade or Ameritrade as it would be my
livelihood.  If I were an eBay seller, why would I care about YouTube?  You
get the idea.  What makes Google, YouTube, Yahoo, MS, etc more important?  

More importantly, why is PCCW not prefix filtering their downstreams?
Certainly AS17557 cannot be trusted without a filter.

Randy

> -Original Message-
> From: Simon Lockhart [mailto:[EMAIL PROTECTED] 
> Sent: Sunday, February 24, 2008 2:07 PM
> To: Tomas L. Byrnes
> Cc: Michael Smith; [EMAIL PROTECTED]; [EMAIL PROTECTED]; 
> nanog@merit.edu
> Subject: Re: YouTube IP Hijacking
> 
> On Sun Feb 24, 2008 at 01:49:00PM -0800, Tomas L. Byrnes wrote:
> > Which means that, by advertising routes more specific than the ones 
> > they are poisoning, it may well be possible to restore universal 
> > connectivity to YouTube.
> 
> Well, if you can get them in there Youtube tried that, to 
> restore service to the rest of the world, and the 
> announcements didn't propogate.
> 
> Simon
> 

RE: Sicily to Egypt undersea cable disruption

[Randy Epstein]

RodBeck said:

>Telecommunication facilities have rarely been targets of terrorism. There
is only one known case - the Tamil Tigers destroyed a central office in Sri
Lanka some years back. My guess is that terrorists want to kill people, not
destroy optical muxes, >Class 5 switches, and the like.

Actually, last year, Scotland Yard claimed Al Qaeda planned on blowing up
one of the Telehouse facilities in the UK:
http://www.technologyreview.com/blog/garfinkel/17561/

Randy

RE: router install in Troy, Michigan

[Randy Epstein]

Craigslist is > that way.

 

  _  

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dorn
Hetzel
Sent: Saturday, October 06, 2007 4:49 PM
To: nanog list
Subject: router install in Troy, Michigan

 

 

apologies if this is non-operational content.

 

I have a customer site in the Troy, Michigan area where I need a small
(Cisco 2610) router installed next week.

 

If you live/work in the area and would like an hour or two of extra work,
please email me back with your contact information.

 

It's a customer site, so you would need to be presentable and professional,
but it's a simple task (one T1, one ethernet, one power cord) and call me
for testing.

 

Regards,

 

Dorn Hetzel

 

RE: i think the cogent depeering thing is a myth of some kind

[Randy Epstein]

> at  there is a plain text document
> with
> the following HTTP headers:
> 
>   Date: Fri, 28 Sep 2007 21:56:34 GMT
>   Server: Apache/2.2.3 (Unix) PHP/5.2.3
>   Last-Modified: Fri, 28 Sep 2007 19:15:53 GMT
>   ETag: "92c1e1-a85-43b36ea5bcc40"
>   Content-Length: 2693
>   Content-Type: text/plain
> 
> the plain text title is:
> 
>   Cogent shows hypocrisy with de-peering policy
> 
> the plain text authorship is ascribed to:
> 
>   Dan Golding

Clearly you can see the article was published by T1R in their Daily T1R
report: http://www.t1r.com/

(listed under "The Daily T1R Headlines")

If you subscribe to the Daily T1R, you can find Dan's report issued today.

> since i appear to be reaching the aforementioned web server by a path that
> includes cogent-to-nlayer, i think this part of the plain text is
>inaccurate.

I think Dan overstepped here.  Richard has made comments of a de-peering
notice received by nLayer, not an actual de-peering occurrence.

AFAIK, the only two networks in recent weeks that have been de-peered are WV
Fiber and LimeLight.  WV was de-peered a couple on September 17th and
LimeLight was de-peered yesterday.

Randy

RE: Cogent issues in SF area?

[Randy Epstein]

Maybe they depeered themselves.  They seem to be on a roll!

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> Mike Lyon
> Sent: Friday, September 28, 2007 2:39 PM
> To: NANOG
> Subject: Cogent issues in SF area?
> 
> 
> Anyone else seeing it?
> 
> BGP_Level3>traceroute 208.70.27.35
> 
> Type escape sequence to abort.
> Tracing the route to 208.70.27.35
> 
>   1 4.79.220.77 0 msec 4 msec 0 msec
>   2 4.68.123.30 [AS 3356] 8 msec 0 msec 4 msec
>   3 4.68.18.5 [AS 3356] 0 msec 4 msec 0 msec
>   4 4.68.110.138 [AS 3356] 4 msec 0 msec 4 msec
>   5 154.54.6.81 [AS 174] 4 msec *  0 msec
>   6 154.54.6.133 [AS 174] 4 msec 4 msec 4 msec
>   7 154.54.24.38 [AS 174] 4 msec 4 msec 4 msec
>   8  *  *  *
>   9  *  *  *
> 
> Bah!
> 
> -Mike

RE: Why do we use facilities with EPO's?

[Randy Epstein]

> FWIW, do you imagine that's terribly large for urban firefighters in
> the big scheme of things, not just computer rooms?
> 
> My memory could be wrong but I remember the John Hancock building, 60
> stories, pulls about 1.5MW...I remember Boston Edison mentioning this
> in discussing a design I was working on of a supercomputer facility,
> that we were asking for more power than the hancock building which was
> ok but it presented..."challenges". Factories can pull a lot of power
> also (that room was never built.)
> 
> Anyhow, once you're beyond a pea-shooter I don't think procedures for
> firefighting vary a whole lot, other than some outliers.
> 
>   -b

I guess my point was that it's safer to power off a UPS system as best you
can before you shoot water at it.  :)  Most likely you are doing this at
somewhat close proximity, with step-down transformers nearby, etc.

An EPO not only shuts down the power feed to the UPS, but the UPS as well.
Which is a good thing.

A properly placed EPO and warning signs, as well as proper training of your
customers and vendors should minimize the risks associated with an EPO.

Look, if someone is hell bent to destroy your facility, EPO or not, they
will succeed.

Randy

RE: Why do we use facilities with EPO's?

[Randy Epstein]

(snip)

> Put another way: Between a 120KVA UPS and a gang of experienced
> firefighters with charged hoses I'd put my money on the firefighters
> every time.
> 
> --
> -Barry Shein

You realize the UPS systems we're speaking of are much larger?  Usually 480
volt, many kVA.

Randy

RE: Routing public traffic across county boundaries in Europe

[Randy Epstein]

Andy,

I've always wondered this as well.  Similar scenario, although not
necessarily egress in a foreign country, but transiting through.

For a brief period, we had an OC48 that carried packets on our network
between Chicago and Seattle that traversed a router of ours in Vancouver, BC
Canada.

Any legal minds here that may know the answer?

Randy

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> Andy Loukes
> Sent: Thursday, July 26, 2007 3:53 AM
> To: nanog@merit.edu
> Subject: Routing public traffic across county boundaries in Europe
> 
> 
> I think this is a pretty dumb question, because I presume this is how
> most organisations save money and provide resilience.
> 
> What (if any) are the legal implications of taking internet destined
> traffic in one country and egressing it in another (with an ip block
> correctly marked for the correct country).
> 
> Somebody mentioned to me the other day that they thought the Dutch
> government didn't allow an ISP to take internet traffic from a Dutch
> citizen and egress in another country because it makes it easy for the
> local country to snoop.
> 
> I've done lots of searching and have our legal council investigating but
> I thought someone here might be able to point me in the direction of any
> legislation?
> 
> (I'll summarise any off-list replies)...
> Thanks,
> --
> Andy Loukes
> 
> Senior Systems Architect
> The Cloud Networks
> http://www.thecloud.net/content.asp?section=1&content=32

RE: Cogent Peering

[Randy Epstein]

Keith,

I believe he meant he would like to purchase transit from Cogent.

-Randy

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> keith
> Sent: Monday, May 14, 2007 2:53 PM
> To: Kevin Billings
> Cc: nanog@merit.edu
> Subject: Re: Cogent Peering
> 
> 
> Do you not know what your traffic ratios are with Cogent? You can easily
> get this information using Sflow or Netflow.
> 
> Keith O'Neill
> Pando Networks
> 
> 
> Kevin Billings wrote:
> > Can someone tell me if there are any tools on the net we can use to
> > evaluate Cogent as a possible Tier 1 peer.  We are looking at adding a 1
> > or 2 Gig connection to them, but after reading some of the posting I am
> > not sure this would be a wise move.
> >
> > Kevin Billings
> > Sr Network Engineer
> > Spirit Telecom
> > 1500 Hampton St
> > Columbia SC
> > http://www.spirittelecom.com
> >

RE: Omaha, NE Carrier Hotels???

[Randy Epstein]

Robert,

Seems like Co-Sentry has a somewhat large facility in Omaha
(http://www.cosentry.com).

First National has one as well.
http://www.fntsinc.com/pdf/Omaha-stat-sheet-06.pdf

Also, although not carrier neutral, I've been to the AT&T building in Omaha
and they do provide co-location services (albeit for their own customers,
other carriers do have connectivity to this facility.)

Regards,
Randy

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> Robert Boyle
> Sent: Wednesday, May 09, 2007 10:17 AM
> To: nanog@merit.edu
> Subject: Omaha, NE Carrier Hotels???
> 
> 
> 
> Omaha is right in the middle of the US and it seems to be a point on
> most carriers' national backbone maps. There has to be some type of
> carrier hotel there somehere, but I can't seem to find it. Can anyone
> provide insight on the 60 Hudson or One Wilshire or 111 8th or Westin
> of Omaha? Thanks!
> 
> -Robert
> 
> 
> Tellurian Networks - Global Hosting Solutions Since 1995
> http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
> "Well done is better than well said." - Benjamin Franklin

RE: Interland dead?

[Randy Epstein]

> Subject: Interland dead?
>  
> Anyone know what's going on?

Wasn't some portion of their assets acquired by Peer 1?

-Randy

RE: Cable Tying with Waxed Twine

[Randy Epstein]

Hey Marty :)


> and digg it:
> 
> http://www.digg.com/mods/The_lost_art_of_cable-lacing...

Corrected URL:
http://www.digg.com/mods/The_lost_art_of_cable-lacing...?cshow=194773

> -M<

Randy

RE: Anything going on in Atlanta, GA?

[Randy Epstein]

Bill,

> Switch and Data was reporting power issues at 56 Marietta
> earlier.  Don't know if it was isolated to their suite, or
> more widespread.
> 
> bill

No issues on 2nd, 3rd or 4th floor.  Not sure about the 6th (where S&D is
located.)

There are also separate generators in the building for the various tenants.

Regards,

Randy

RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted

[Randy Epstein]

>   I've wondered how many boats/subs exist for these repairs
> and if attempting to do them all in parallel is going to be a big
> problem.  With 6 systems having outages, it will be interesting to see
> when various paths/systems come back online and if there is a gating
> factor in underseas repair gear being available in the region.

Just to give you an idea:

(from
http://www.cnn.com/2006/WORLD/asiapcf/12/27/taiwan.quake.ap/index.html) 

(c)2006 AP

Tyco International Ltd. said it has a Taiwan-based cable-laying ship heading
to the area for repairs.

"Pretty much everything south of Taiwan has been reported at fault," said
Frank Cuccio, vice president of marine services at Morristown, New
Jersey-based Tyco Telecommunications.

Cuccio expects the ship to be in position in a few days. It then takes three
to five days to repair each cable, but mudslides set off by the earthquake
can complicate matters by covering the cables, making them harder to
retrieve from the bottom.

Cuccio said the ruptures are more than 10,800 feet below sea level, too deep
for the remote-controlled submersibles that otherwise would find the cables.
Instead, the ship will drag grapnels along the bottom to find them.

The cables on the deep ocean floor are just two-thirds of an inch, a
testament both to the immense data capacity of optical fiber and the
fragility of the links that form the global telecommunications network.

> 
>   - jared

Randy

RE: Collocation Access

[Randy Epstein]

> AT&T's colocation facility in mid town retains your ID. So do a lot of
> others I've been to. And that happens whether or not they give you a cage
> key.

Maybe this is a recent "feature".  From what I've seen, AT&T's security
policy differs from site to site, employee to employee, no matter what they
claim.

> -Don

Randy

RE: Power issue at Telx NYC?

[Randy Epstein]

Drew,

There is definitely a power outage at telx/60 Hudson.  According to telx,
this was a scheduled maintenance gone bad.  I have someone onsite and he is
reporting that power should be restored shortly.

Note, those with redundant feeds within the facility should only see a
partial outage.

Regards,

Randy

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
> Drew Linsalata
> Sent: Saturday, December 02, 2006 1:09 AM
> To: nanog@merit.edu
> Subject: Power issue at Telx NYC?
> 
> 
> 
> One of our transits is reporting a power outage at Telx in NYC tonight.
>   Does anyone else have any reports of a power problem in that facility?
>   We're trying to get some word from Telx now too, but they're slow to
> respond.
> 
> 
> 
> 
> 

RE: Collocation Access

[Randy Epstein]

>Then you broke the law, assuming you had a Florida license and you
>presented to the Miami facility.

Actually, I handed them an Austrian license.  Maybe I violated some EU
directive! 

>DS

Randy

RE: Collocation Access

[Randy Epstein]

> From what I've seen, there's a complete lack of awareness of the 
>risks associated with retention of identification or information. I 
>even had a long argument with the local US Post Office, who wanted to 
>record numbers from two forms of ID in order for me to retain my PO 
>Box. Their claim was that postal inspection service requires it. I 
>objected due to my local postoffice storing this information on index 
>cards which all employees of the post office can access. While I 
>understand the postal inspection service's interest in being able to 
>track down box holders, I asked the postmaster if he'd sign a 
>document accepting personal responsibility if the information was 
>released or used by any of his employees.

  .. and how did that go?

>I think it's time to show up with such a statemant of acceptance of 
>liability whenever asked for such information. I have to wonder if 
>company lawyers would then give it some thought. 

  Being recently on a large, well known military station, the opposite
happened to me.  While yes, when originally being vetted I had to supply
certain information that most would cringe at supplying, when onsite I was
asked for two forms of government issued identification (I chose drivers
license and passport) which was just reviewed (not copied), immediately
handed back to me and then asked to pose for a picture and signed an
electronic pad.  A minute later I was handed a new government issued ID.
During my stay, I had the need to access certain restricted areas.  As I
entered restricted area buildings, I was handed a restricted area badge to
wear over my new picture ID to let people know immediately what areas I had
access to (the alternative is shoot first, ask questions later; I'll pass,
thanks).

  On the other hand, I've visited many data center, collocation facilities,
and even foreign military bases (both US and others), and since AT&T sparked
this conversation, I've actually been to nearly 40 of their facilities
throughout the US.  In recent memory, I can think of two large collocation
centers that retain your ID.  One is in Miami and one in New York (I don't
think I need to name names, most of you know to which I refer).  All others
(including AT&T) have never asked to retain my ID.

  I'm not exactly sure why these sites want to retain ID, but I think it
goes along with the big weight that is connected to the gas station bathroom
key.  They want to make sure you return your cabinet keys (if any),
temporary pass (if any), etc.  Legal risk or not, can you think of a better
way to get someone to return to the security desk to sign out?  Until then,
these sites will continue this practice.

Randy

RE: Bandwidth accounting recommendation?

[Randy Epstein]

Hello,

>   Hi, I have been scouring the net searching for a good bandwidth
>accounting solution that would be appropriate for a hosting
>provider/carrier. We are more interested in the total amount of
>bandwidth the user has utilized in a 7/30/90/365 (whatever) day period
>of time than a Mbps 'graph' which MRTG would give you. It would also be
>great if it could allow us to assign logins to our users so they can
>view their utilization.

If you have a budget put together for this type of application (you'll need
it!), Orion from Solarwinds (http://www.solarwinds.net) would suit your
needs.  I have used Orion for over 2 years now and quite satisfied with its
features and performance.

>   So far I've looked at MRTG, Cacti, and RTG. Cacti was pretty
>good execept it doesn't appear to notice changes in a switch, sometimes
>more than 30 ports on 5 different switches change a day and we'd like
>something that automatically starts/stops monitoring utilization when
>the port status changes. I havent found a Netflow tool yet that I really
>like.

I don't fully understand your requirements here, but maybe the folks at
Solarwinds can provide you with a solution here.

>Any suggestions?

>Thanks,
>Andrew

Regards,

Randy Epstein

Email: repstein(at)chello.at

RE: ICG Experience

[Randy Epstein]

Aaron Glenn wrote:

>if by "assimilated rather quickly" you mean "answers the phone with
>'Level3' instead of 'Wiltel' " then yes, they are. Otherwise it's the
>same network with the same equipment and generally the same people
>pre-acquisition

I would think that Elijah was looking slightly longer term than
pre-acquisition.  Based on when Level(3) made the acquisition announcement,
I'd guess that they are fairly close to completing it, and if you've noticed
the changes at some of their other recent acquisitions, they typically do a
lot more than just change the way they answer the phone.

As far as Wiltel goes, the entire sales team I was dealing with was gone by
the time the deal completed.  There were also entire outside plant personnel
changes, etc.

I've also recently dealt with Level(3) acquiring other vendors of mine, such
as Progress Telecom and Telcove.

I stand by my statement, but opinions of your experiences are welcome of
course.
 
-Randy

RE: ICG Experience

[Randy Epstein]

Elijah Savage wrote:

 





>Hopefully this will be my last time querying the group for
provider experience. My previous experience with them was a while back when
they filed for bankruptcy and cut back on support, but a coworker just informed
me they have since been purchased by Level3. Is there anyone here that has any
cross >connects or any type of connectivity to
them and wish to share their experience offline I would appreciate it.



When a company is acquired by Level(3),
they are assimilated rather quickly.  The company you would be dealing with is
Level(3), so make your decisions based on that.

 

-Randy

RE: AT&T routing

[Randy Epstein]

Andrew:

> Would a routing engineer from AS7132 (SBC/AT&T) please contact me off 
> list to resolve a routing issue I've discovered on your network?

> Andrew D Kirch  |   Abusive Hosts Blocking List  | www.ahbl.org
> Security Admin  |  Summit Open Source Development Group  | www.sosdg.org

Your email address bounces and the phone number listed in your whois record
for trelane.net is disconnected.  Is there a message you'd like to pass
along to an engineer?

Contact me offlist.

Regards,

Randy

RE: Global Crossing Contact / BGP and SONET interaction question

[Randy Epstein]

Forrest:



>Recently my BGP session has started flapping on the GX circuit... It 
>looks something like this:
>
>Jul 21 21:33:32.703 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
>
>There are no other log entries during the periods when this occur. 
>Unfortunately this causes enough prefix flaps that any prefixes which 
>are preferred through GX are damped for like a half hour by certain 
>providers as my BGP routes get added/withdrawn through the GX link.



I don't have an answer to the root cause of your problem, and I'm not
looking for a discussion on route dampening (there are enough debates on
this issue to make your head spin), but may I suggest you raise your hold
timers to prevent your BGP sessions from going down on short disturbances as
these?

>-forrest

Randy