RE: Level 3, again

[David Hubbard]

Just got confirmation from them that a master ticket is
being created and the transport team is looking at it.
There's an issue with an EBR router in Dallas that
they have already routed around but from what I can see,
there are still major issues with traffic that has to
traverse Dallas.  Our connection to L3 in Atlanta going
to the same destinations has no issue but jumps from
ATL to DC to LAX.

David 

> -Original Message-
> From: Chris Ranch [mailto:[EMAIL PROTECTED] 
> Sent: Tuesday, February 12, 2008 1:32 PM
> To: David Hubbard; nanog@merit.edu
> Subject: RE: Level 3, again
> 
> This affected us too, same impact.  No official word from 
> Level3, but it
> looks to be back up now.  
> 
> My normal path was SJO-LAX-DAL-MIA, now it's 
> SJO-LAX-DEN-DAL-MIA.  I can
> only speculate that it was related to the LAX-DAL link.  It 
> lasted about an
> hour.
> 
> Chris 
> 
> > -Original Message-
> > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> > Behalf Of David Hubbard
> > Sent: Tuesday, February 12, 2008 10:06 AM
> > To: nanog@merit.edu
> > Subject: Level 3, again
> > 
> > 
> > Anyone know what's going on on their network?  We opened a
> > ticket but haven't heard back, sounds like they may have
> > some kind of nationwide issue going on that started in
> > Atlanta.  We've had customers on Time Warner, SBC Global,
> > AT&T and Pac Bell unable to reach us.
> > 
> > David
> > 
> 
> 
> 

Level 3, again

[David Hubbard]

Anyone know what's going on on their network?  We opened a
ticket but haven't heard back, sounds like they may have
some kind of nationwide issue going on that started in
Atlanta.  We've had customers on Time Warner, SBC Global,
AT&T and Pac Bell unable to reach us.

David

RE: Level 3 (3356) issues?

[David Hubbard]

Looks like this was localized to Tampa.  I've received
emails from two other people connected through Tampa, like us,
who were having the same issues.  I finally got TCAM on the
phone after about an hour.  They have a master ticket for a
failure of "three DLM modules" lasting 47 minutes but it is
showing believed to be resolved via reset but they have not
yet diagnosed the cause.

David

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of David Hubbard
> Sent: Tuesday, January 15, 2008 11:45 AM
> To: nanog@merit.edu
> Subject: Level 3 (3356) issues?
> 
> 
> Just curious if anyone is seeing issues with Level 3
> right now?  Our session is still up but we can't 
> see any outside routes through them currently.  I'm
> guessing by the fact that I've been on hold for 25
> minutes that I'm not the only one having an issue with
> them but wanted to double check.
> 
> Thanks,
> 
> David
> 
> 

Level 3 (3356) issues?

[David Hubbard]

Just curious if anyone is seeing issues with Level 3
right now?  Our session is still up but we can't 
see any outside routes through them currently.  I'm
guessing by the fact that I've been on hold for 25
minutes that I'm not the only one having an issue with
them but wanted to double check.

Thanks,

David

Abusive traffic from Microsoft China?

[David Hubbard]

Just wondering if anyone else is seeing huge random
floods of traffic from:

inetnum:  202.96.51.128 - 202.96.51.255
netname:  MICROSOFT-CO
descr:Microsft (China) Co.Ltd
country:  CN
admin-c:  CH455-AP
tech-c:   SY21-AP
mnt-by:   MAINT-CNCGROUP-BJ
changed:  [EMAIL PROTECTED] 20060926
status:   ALLOCATED NON-PORTABLE
source:   APNIC
changed:  [EMAIL PROTECTED] 20060926

On a nearly daily basis we see them randomly open
thousands of connections from a variety of addresses
in that block to multiple servers.  I've emailed
of coruse but that results in nothing.  Probably
will just end up blocking them.

Thanks,

David

RE: Least Sucky Backbone Provider

[David Hubbard]

From: Gregory Boehnlein
> 
> Good morning,
>   I'm considering dropping Cogent completely out of my 
> transit mix, as the number of outages and problems they
> have been experienced over the past year has reached an
> unacceptable level. It has gotten to the point that we
> their BGP session is shutdown for longer periods than it
> is on. Based on the availability of on-net fiber in my
> facility, I have narrowed the field to the following
> candidates:
> 
> 1. Level 3
> 2. MCI/Verizon
> 3. AT&T
> 
> I'm looking for comments from actual customers of the above 
> providers in relation to;
> 
> 1. Network reliability and performance
> 2. Responsiveness to outages
> 3. Proactive notification of network maintenance
> 
> 95% of our traffic mix is US48 in nature, so International 
> routes are not a huge decision point.

We've got connectivity to all three of those plus Cogent
and TWTC.  Similar to what someone else posted, a much smaller
percentage of our web hosting traffic seems to naturally prefer
the other providers compared to Level 3.  In some cases this
has appeared to be forced as testing has shown other paths
to be optimal in speed/delay, so we've forced traffic to some
ISP's out through other providers manually rather than going
over the L3 link.  We still send the bulk of the traffic over
L3 though as they are optimal in many cases and very reliable.

In order of reliability, it's been Level 3, AT&T,
UUNet, TWTC, Cogent with all providers being in use with us for
at least 18 months.  Level 3 and AT&T have always been
very responsive and I've had success getting someone who knows
BGP rather than a delayed call back.  Never actually called
UUNet since the few outages there have been have all been short
enough that we just disabled the session for a bit and brought
it back up with no issues so I can't say with experience how
they actually are, although since I have never needed to call
that is a good thing.  If you're single homed probably best to
find out more on their support if you were going to use them.

TWTC I've had some issues with community filtering rules that
we need dropping out with no warning and multi-hour delays
getting someone who could help.  Cogent, with their unusual
inbound/outbound peering setup, seems to have an issue with
their outbound side of our link regularly, every few weeks,
while the inbound session has been up for 460 days.  I also
dislike their very poor community offerings, we can't do
many of the things with them that we do with the others.  We
keep them and prepend three times on our advertisements to them
so they are primarily just a link for traffic going to and from
other Cogent customers.

If the pricing is right, I think you'd be happy with either
L3 or AT&T, probably MCI/Verizon/UUNet (or whoever owns them
this week) too provided their support is good since we have
found them to be pretty reliable.

Dave

Level 3 Colo question

[David Hubbard]

Question for you all; does anyone have experience with Level 3's
colo offerings, and if so, have your prices increased dramatically
for power and square footage as contract renewals have come around?
Do you know if they have a practice of pricing out customers they'd
prefer to have move elsewhere?  Would like to compare notes 
privately if so...

Thanks,

David

RE: Does anyone multihome anymore?

[David Hubbard]

From: Mike Tancsa
> 
> Hi,
> In my case, I have 6453 and 174 for transit.  I want to get to 577 
> which is directly connected to 6453 and 174. 577 has a higher local 
> pref on paths via 174.  Short of shutting my 174 session (or some 
> deaggregation), I dont have a way to influence how 577 gets back to 
> me.  I can easily exit out 6453, but it does nothing for the return 
> packets.  I have enough capacity on 6453 to handle all my traffic, 
> but its a Draconian step to take and some traffic via 174 is fine and 
> would be worse if I fully shut the session. (ie. peers of 174 
> in Toronto)

That's because Cogent has chosen to not give us the BGP communities
needed to influence traffic destined to you from their other peers.
They're one of the providers I use and I just turn them off when they
are having issues.  With Level 3, for example, I can path prepend to
just certain remote AS's, etc., it's nice to have that control.

Dave

RE: Level3 / Qwest routing issues earlier today?

[David Hubbard]

From: Chris Riling
> 
> Hi Guys,
>  
>  Was anyone else seeing anything weird going on today? I 
> have an OC-3 to Qwest, and another OC-3 to Level3, (among 2 
> others to different providers) and when all was well, I was 
> receiving a little over 221,000 prefixes from L3. Then, 
> intermittently I would start losing prefixes from L3 and it 
> would get down to a little over 220,000. When this happened, 
> I noticed I was having intermittent connectivity issues 
> (atleast to one IP I was trying to get to). Doing a 
> traceroute during the "outage" the packets would hit Qwest's 
> network, then L3's, then drop, or sometimes not. I have since 
> admin downed my BGP peer to L3 and everything  has 
> stabilized, but I was wondering if anyone else saw anything going on? 

I think there was something going on between Qwest and L3,
I saw the same thing with traffic coming from a network I
have equipment on which is connected with Qwest and tracing
to our home AS which has L3 amongst others.  I set a
community for L3 to not advertise my prefixes to Qwest to
get around it.  It appears back to normal now though, I
just tested.

Dave

TWTC issue with Foundry routers?

[David Hubbard]

Anyone know of any changes that were made with TWTC (AS 4323)
last night that may have affected those running Foundry
routers?  We peer with a number of providers and last night
our TWTC connection went down with:


Jul 25 15:57:22:N:BGP Peer 1.2.3.49 DOWN (Attribute Flags Error)
Jul 25 15:57:14:N:BGP Peer 1.2.3.49 UP (ESTABLISHED)

If I debug updates on that session I get:
(Lines added for readability)


Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 142.166.102.0/24
Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP
AS_PATH=AS_SEQ(2) 4323 8881 8881 8881 30915 NextHop=1.2.3.49
COMMUNITY=4323:51 4323:501 4323:1003 4323:2001 4323:2503 4323:34510
4323:5 65101:1003 65102:4 65103:1 65104:301 
Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 193.27.220.0/23
Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP
AS_PATH=AS_SEQ(2) 4323 2828 19092 14188 14188 14188 14188 14188
NextHop=1.2.3.49 COMMUNITY=4323:51 4323:501 4323:1015 4323:2503
4323:36410 4323:5 65101:1015 65102:4 65103:1 65104:301 
Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 64.13.0.0/22


Jul 25 15:57:21 BGP: 1.2.3.49 rcv invalid COMMUNITY attribute flag d0


Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP
AS_PATH=AS_SEQ(2) 4323 12956 3352 NextHop=1.2.3.49 ATOMIC_AGGREGATE
AGGREGATOR AS=3352 Speaker=81.46.63.133 



The router is a Foundry NetIron 400 running their 7.8 code.
We have two of these talking to Level 3, TWTC, Cogent, Uunet
and AT&T and only the TWTC had an issue.  They sent me a
default route instead of full routes and the session came
up and was stable; go back to full routes and error.  They
admitted to me this afternoon that three other customers are
having the same issue.  That's when we started wondering if
they changed something that the Foundry code doesn't like.
Interesting though is that they claim to not be sending me
communities while the output above indicates they are.

Any ideas; be nice to get the link back up. :-)

Thanks,

David

RE: Level(3) faux paux

[David Hubbard]

From: [EMAIL PROTECTED]
>
> Or maybe past experience has shown that the two have the same 
> *actual* failure rate, but asking for a Level3 help is more
> likely to actually get you a clueful *and* helpful engineer.

This has been my experience when we've had issues with any of
the five companies we purchase from; Level 3 has consistently
been able to put someone on the phone that actually knows how
to resolve problems without even needing to call me back; most
of the others we turn our link off because I know it's going to
be several hours.

David

AS701 route server?

[David Hubbard]

Anyone know of a route server in AS701?

Thanks,

David

RE: AOL Postmaster?

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> 
> 
> >Anybody from AOL on this list? Could you please send me an email
> >offlist?  I need some help.
> 
> Have you pursued every avenue of contact listed at:
> ?
> 
> I've found them to be GENERALLY pretty responsive on those channels, 
> as have many others.
> 
> --chuck

I'm still getting feedback on netblocks we haven't been
associated with in several years and I've tried about
20 times to get them to stop it but cannot.  If you call
they just tell you to email, if you email you get nowhere.

David

Road Runner (as10994) NOC contact?

[David Hubbard]

All of their listed contact info is for abuse and
that just gets you a voice greeting that tells you
to email abuse and then hangs up on you.

Trying to troubleshoot an issue between Road Runner
Tampa and Level 3.

Thanks,

David

RE: comcast spam policies

[David Hubbard]

From: [EMAIL PROTECTED]
> 
 
> The current comcast policy  seems to be to backhole mail 
> servers at random.

This is true.  We do mostly ecommerce hosting and often our
customers have their own dedicated servers blacklisted from
sending email notifications of new orders to their home
comcast addresses.  We have to request unlisting regularly.

David

Contact for AS18566 (covad)?

[David Hubbard]

Tried their support line but they won't talk to me without
a circuit ID or other customer credential and I'm having a
routing issue getting to a portion of their network through
the hand off from Level 3 Miami to them.

Thanks,

David

RE: IP adresss management verification

[David Hubbard]

From: chuck goolsbee
>

> ** I assume it is myth, but I've never heard anyone from Google make 
> any statements that definitively debunks it. Debunking this pervasive 
> among webmasters and "SEO Experts" myth sure would be a very UN-evil 
> thing to do if true (Hint hint you Google-folk!)
> 

Actually they did, once, in an obscure place:

http://interviews.slashdot.org/interviews/02/07/03/1352239.shtml

Whenever I refer customers to that, they just tell me oh,
that's from 2002, my SEO person knows that Google penalizes
for sharing IP's now.

I guess Google needs a time stamped banner on their site
that says "As of , we do not penalize for sharing IP's."
so SEO 'experts' can't argue it.

David

RE: IP adresss management verification

[David Hubbard]

From: [EMAIL PROTECTED]
> 
> SSL is a technical justification for separate IP
> addresses for web hosts. Virtual servers is another
> technical justification for assigning multiple IP
> addresses to a single physical server.

What I meant was we require a technical justification to
give a dedicated IP to a customer but many hosts do not, 
or they use it as a revenue add by charging for having
a dedicated IP when there's no technical reason for it.
Previously, or maybe still, there was no mandate that web
hosts only assign dedicated IP's when it can be justified.

David

RE: IP adresss management verification

[David Hubbard]

From: [EMAIL PROTECTED]
> 
> It's quite simple, really.
> 
> They ask for it.
> 
> If the iformation that you provided with your application
> does not answer their questions, they ask you for more
> information. I assume that all the RIRs will sign an
> NDA with you, certainly ARIN does this. ARIN may also
> ask for corporate confidential information in order to
> verify your application so they have strict internal
> security policies to keep it confidential. 
> 
> Some people send detailed network diagrams, purchase
> orders for routers/switches/circuits, sales history
> data with projected trends, customer lists, etc.
> 
> If you need specific details, just ask your RIR.
> 

Is the policy still that dedicated IP addresses for web
hosts *should* only be used when technical justification
exists?  I really wish it would change to a requirement
as we very frequently get new hosting customers who get
angry when they find their site that doesn't have SSL
or any other technical reason for a dedicated IP ends up
on a shared IP when their old host didn't do that or
would sell them a dedicated IP for $5/month, etc.

David

Verizon PSTN continued

[David Hubbard]

The thread yesterday didn't seem to get into much
detail; I'm wondering if anyone knows more about what
is going on with Verizon?  Our PSTN service with XO
seems to be affected again by what XO claims is a
Verizon problem but they wouldn't elaborate on why they
feel that to be the case; I was just amazed to even get
someone on the phone at XO since normally they are
unreachable.  I am inclined to partially believe them
since I've found other numbers that I know to be with
other carriers that are not working.  XO claimed this
was at least a regional issue of Verizon's that is
affecting multiple carriers.

David

Any issues with AS 19548 and their links to Level 3 or TWTC?

[David Hubbard]

Having some connectivity issues with multiple
customers on that network from our AS and a few
others I've found on traceroute.org; is anyone
aware of anything there?  Traces in, but which
are more likely failing on the return side,
often stop at ae-1-0.c1.dfw91.twc-core.net and
paix-atl.adelphiacom.net.  I've been unsuccessful
trying to make them prefer a different inbound
route to us.

Thanks,

David

AT&T refuses to provide PTR records?

[David Hubbard]

Anyone familiar with AT&T's policies on PTR records
for their customer-assigned address space?  We have a
customer whose website we host that has their own
in-house mail server that they run off of their AT&T
internet connection at their office.  We handle the
DNS for their domain name.  AT&T is refusing to set
up PTR records for them because they're not handling
DNS for the domain name.  Is this normal?  I haven't
dug through the ARIN agreements but I thought it was
required to provide reverse DNS on your allocations.

Thanks,

David

Anyone with Earthlink available to troubleshoot?

[David Hubbard]

Just curious if someone within Earthlink can
contact me or test connectivity to AS 33260;
traces in both directions stop at the Earthlink
border.  Tried calling the number listed with
arin but the operator there only knows how to
route calls if it's a DSL problem.

Thanks,

David

RE: XO outages?

[David Hubbard]

From: Tom Beecher [mailto:[EMAIL PROTECTED] 
> 
> As much as I hate to ever defend that abomination that is XO, 
> I can get to my colocated stuff in Nashville with no problem. 
> Could be Tampa specific, I am hitting XO in DC, running down 
> to Atlanta, and back up to TN without a problem.


Yep, appears that this was an XO metro fiber cut in Tampa.
Evidently the redundant routes weren't.

Thanks to all that sent me info offlist,

David 

> David Hubbard wrote:
> > Anyone know what's going on with XO?  Seem to be having some big 
> > outage in the south east for phone and data; traces to equipment in 
> > Tampa stop at Virginia.  Their customer support lines are 
> as useless 
> > as usual with their 'unusually long hold times' which seem 
> to be quite 
> > predictable at 60 minutes at a minimum every time I have 
> ever called 
> > them.
> >
> > David
> >   
> 
> 
> 

XO outages?

[David Hubbard]

Anyone know what's going on with XO?  Seem to be having some
big outage in the south east for phone and data; traces to
equipment in Tampa stop at Virginia.  Their customer support
lines are as useless as usual with their 'unusually long 
hold times' which seem to be quite predictable at 60 
minutes at a minimum every time I have ever called them.

David

Contact for optonline.net routing?

[David Hubbard]

Anyone have a contact for optonline?

Thanks,

David

Routing issues with cox?

[David Hubbard]

Anyone know of something going on with Cox currently?
We've had a number of customers report they can't get
to us (AS 33260) from Cox (AS 22773).  We've tried
forcing traffic out different providers of ours and it
usually seems to die with Level 3 in DFW, Time Warner
Telecom in DFW, Cogent in Atlanta, AT&T in Atlanta,
etc.  On the Cox side, going on traceroute info from
customers, it looks like they might force their
outbound on Cogent because everything from them seems
to come through that peer even though we prepend that
one heavily.

Thanks,

David

How to tell if something is anycasted?

[David Hubbard]

So I'm looking at a company who offers anycasted DNS;
how do I tell if it's really anycasted?  Just hop on
different route servers to see if I can find different
AS paths and then do traceroutes to see if they suggest
the packets are not ending in the same location?
>From my routers' perspective I don't see a difference,
but then I don't think I should, correct?

Thanks,

David

RE: AOL 421 errors

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> 
> 
> We've noticed a surge in 421 e-mail errors from AOL.
> 
> Message soft bounced for '[EMAIL PROTECTED]', '4.3.2 - Not 
> accepting messages at 
> this time ('421', [': (DYN:T1) 
> http://postmaster.info.aol.com/errors/421dynt1.html', 'SERVICE NOT 
> AVAILABLE']) []'
> 
> It seems as though they've tightened down their policies.
> We're pretty good at preventing spam with our IronPort
> anti-spam gateways and internal policies.
> 
> We've also subscribed to their FBL notification service.
> I'm surprised at the types of messages AOL customers consider
> as spam. Anything and everything: university admission acceptance
> notices; instructor class assignments; photos from friends; etc.
> 

AOL users report receipts for things they have bought or
letters from their kids as spam, or just use the spam button
instead of delete.  We have a feedback loop with them on a
netblock we moved off of about 24 months ago; I've emailed
their postmaster at least 12 times with no response and no
change.  I've called and they refuse to talk and say I must
email or use the form on the postmaster site to be removed;
no effect.

David

RE: Open Letter to D-Link about their NTP vandalism

[David Hubbard]

From: Rubens Kuhl Jr.
> 
> 
> 
> It still would require him to answer the DNS requests. Only 
> way to addres that is everybody outside DIX declare 
> gps.dix.de as www.dlink.com in their resolvers.
> 

How about serve back bogus NTP data to non-BIX customer
prefixes?  Maybe if people's computers start setting
themselves to the year 2004 D-Link will do something. :-)

Dave

RE: Foundry Old Switch vs Old Cisco one

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> 
> Dears,
> 
> I have this old foundry switch in the warehouse, I have no 
> experience in Foundry, i wonder if this switch can be 
> upgraded to a newer OS that will support advanced features or 
> shall i consider it dead,
> 
> I want to mainly use it for one customer that wants caching, 
> its L4 i guess and i have an old NetApp caching server that 
> will save the customer 10MBs i guess.

You have a Foundry ServerIron XL:

http://www.foundrynet.com/services/documentation/index.html#SC

David

RE: Honest Cogent opinions without rhetoric.

[David Hubbard]

From: Omachonu Ogali [mailto:[EMAIL PROTECTED] 
> 
> I have the need to de-pref my routes to Level3, to be of 
> equal value as the routes they receive from their peers,
> but they don't offer a community for that. But wow, I can
> see that this route originated from Tustin, CA!

I've found them pretty responsive to unusual requests,
maybe they'd find a way to do that if you ask them; or
go through your sales rep if the support guys aren't
being helpful.  Or maybe you can alter the preferences
your providers that peer with them announce for you.
Besides them, we've also had good help for unusual
things from Global Crossing, who we no longer use for
reasons other than dissatisfaction, and Time Warner
Telecom.  Have not yet had a need to make unusual
requests of our other providers so I can't say how they'd
deal with it.
 
> Everyone's traffic engineering needs are different. BGP 

Yes, that's why the more a provider offers, the better.
I'd rather have more than I need than not having what I
currently need; my only point was that Cogent's community
support is more lacking than the others I have used and
BGP communities are often something that is not considered
until the link is up or until the first problem where they
would come in handy is encountered.

 
> P.S. You have transit, not peering.

Thank you, an egregious typo on my part.

David

RE: Honest Cogent opinions without rhetoric.

[David Hubbard]

From: Charles Gucker
> >
> > Depends on if you like to do traffic engineering; Cogent's
> > BGP community support, consisting of a whole three things you
> > can set (two if you only have a single connection to them),
> > makes that rather difficult.
> 
> Umm, where did you get that mis-information from?   The last
> communitiy guide I've seen from them has quite a bit more than two ;p
> 
> For a quick reference:
> http://www.onesc.net/communities/as174/as174.pdf
> 
> charles

I was meaning more along the lines of types of things you can
control.  You have limited route export control, and even more
limited path prepend control; that's two. :-)  Or if you're
multi-homed to them, you can set local pref too.

Alternatively, I'll just use Level 3 as an example because we
have peering with both of them as well as three others. Level 3
lets us control local pref, prepend to specific remote AS
numbers (incredibly useful and something everyone should insist
on from their providers), prepend to EU peers, all the
same export controls and then some, announce to specific AS,
do not announce to specific AS.  On the incoming side, where
Cogent has tags for all of four whole groups (NA non-customer,
NA customer, EU non-customer, EU customer), Level 3 has tags for
the same plus peering points, cities, countries, regions, MEDs
ignored, MEDs accepted, internal, etc..

We like having our connectivity to Cogent because we like to
have fast delivery of our customers' content to Cogent's
customers, not because of the price.  On more than one
occasion we have had issues where the only solution to a
problem of our customers accessing us coming through a
remote peer of Cogent's who had a problem, and a preference
for the Cogent route, was to turn down our session with them
until that remote peer resolved their issue; there's no other
way to make them prefer a different route, including with
the help, or lack thereof, of Cogent's NOC.

David

RE: Honest Cogent opinions without rhetoric.

[David Hubbard]

From: [EMAIL PROTECTED]
> 
> On Wed, 8 Mar 2006, Martin Hannigan wrote:
> 
> >
> > We have heard a lot of negatives about them, about their
> > pricing model, about their network, about de-peering with 
> > Level 3, etc.  What we really need is actual information.
> >
> Much of the negatives is from jaded competitors who don't 
> want to fairly compete. Other than that, the answer is 'it depends'. 

Depends on if you like to do traffic engineering; Cogent's
BGP community support, consisting of a whole three things you
can set (two if you only have a single connection to them),
makes that rather difficult.

David

RE: Presumed RF Interference

[David Hubbard]

From: Jon R. Kibler

> 
> I should also add some other points:
>-- We have observed failures when the building had zero 
> power, except for the UPS battery power in the server room, 
> so we don't think that we are getting power spikes from 
> anything within the building.

If you had failures before utility power was even applied, it sounds
like multiple building grounds with differences in potential.

David

RE: Fed Bill Would Restrict Web Server Logs

[David Hubbard]

From: Andy Davidson
> 
> 
> Speaking with my e-commerce vendor hat on, server logs (apache, mail, 
> application audit logs) and other information about visitors 
> (especially those who have conducted a purchase transaction with
> us, or signed up to our newsletter) never stop having a business
> purpose - it's called referential integrity.
> 
> We want to use them to track the behaviour fraudulent users 
> for example.

Anyone who runs mailing lists has to keep that info to be
able to prove how and when someone opted in.

David

RE: Problems connectivity GE on Foundry BigIron to Cisco 2950T

[David Hubbard]

You are using a crossover cable right?  If that's all set, you
do need to have neg-off on the Foundry and "no nego auto" on the
Cisco.  I haven't used the rj-45 gbics in the Foundry equipment
before, not sure if that could be an issue.  I would go with
the hard set 1000-full on both sides.

David 

From: Sam Stickland
> 
> Hi,
> 
> I'm having a right mare trying to get a Foundry BigIron to 
> connect up to a cisco 2950T, via Gigabit copper.
> 
> The Foundry BigIron is using a cisco RJ45/copper GBIC that 
> was pulled from a live cisco 6500, where it was working
> fine. The cisco 2950T has two fixed 10/100/1000 RJ45 ports.
> 
> The cables between the equipment have been tested and are fine.
> 
> The Foundry has three different types of the gigabit negiation modes:
> 
>auto-gigAutonegotiation
>neg-full-auto   Autonegotiation first, if failed try 
> non-autonegotiation
>neg-off Non-autonegotiation
> 
> I've tried all three, complete with all the other 
> possibilities with the cisco 2950T (which has fixed full
> duplex operation, but can be set to 'speed auto' or
> 'speed 1000').
> 
> None of these combinations bring up the link. The cisco 2950 
> never gets a link light. The Foundry gets a link light
> regardless when it's mode is set to 'gig-default neg-off'.
> 
> I'm at a bit of a loss to explain this. Does anyone know of any 
> configuration issues that can explain this, or is it time to start 
> swapping out hardware components?
> 
> Sam
> 
> 

RE: Level3 Blackhole Community

[David Hubbard]

Their supported communities are available from their
whois; I don't think blackhole is one of them but 
they do have one that will allow you to suppress your
announcements to remote peers of your choice if that
would help.

David

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of Erich Borchert
> Sent: Wednesday, December 14, 2005 7:51 PM
> To: [EMAIL PROTECTED]
> Subject: Level3 Blackhole Community
> 
> Does anyone know if Level3 supports a BGP black hole 
> community from their customers, .e.g., 3356:666 ?  I spoke 
> with someone in their NOC but they lacked clue.  
> 
>  
> 
> Thanks,
> 
> -Erich
> 
>  
> 
>  
> 
>  
> 
>  
> 
>  
> 
> 

RE: Cogent issues

[David Hubbard]

Anyone else find it funny that of all the Cogent
peering on that chart, only the Cogent to Level 3
link shows green?  :-)

David

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of Ryan A. Krenzischek
> Sent: Thursday, November 17, 2005 10:49 AM
> To: nanog@merit.edu
> Subject: Re: Cogent issues
> 
> 
> Take a look at this:
> 
> http://scoreboard.keynote.com/scoreboard/Main.aspx?Login=Y&Use
> rname=public&Password=public
> 
> I just got off the phone with Cogent.  They said there was a 
> fiber cut in the 
> eastern-US.  The representative did not have specific 
> details.  They said 
> either in the NY or DC area.
> 
> Ryan
> 
> 
> On Thu, 17 Nov 2005, Lyons, Myke wrote:
> 
> > For the past hour or so a number of sites that I have with 
> Cogent have
> > been unreachable.  Also, I am unable to get through to their support
> > line.  Is anyone else seeing this?
> > 
> > .myke lyons
> > 
> >
> 
> 

RE: [NANOG]Cogent issues

[David Hubbard]

They've got something going on, we've got about a
25% drop in our outbound Cogent traffic at about
9:45 AM EST.

David 

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of Lyons, Myke
> Sent: Thursday, November 17, 2005 10:31 AM
> To: nanog@merit.edu
> Subject: [NANOG]Cogent issues
> 
> For the past hour or so a number of sites that I have with 
> Cogent have been unreachable.  Also, I am unable to get 
> through to their support line.  Is anyone else seeing this?
> 
> .myke lyons 
> 
> 

AOL Postmaster contact?

[David Hubbard]

Are there any AOL Postmasters on the list?
I'm having an issue that the toll-free regular AOL
Postmaster helpdesk telling me will take 3-5
business days to resolve, and it's more urgent
then that.

Thank you

David

RE: Level3 problems

[David Hubbard]

Anyone get anything useful out of L3 yet?

Dave

From: John van Oppen (list account) [mailto:[EMAIL PROTECTED] 
> 
> I am getting fast busy signals on all my Washington based 
> level3 DID numbers at the moment...   
> 
> A level3 full peer up here seems to only seek 68k routes... 
> not so good (thankfully that was not on my network).
> 
> 
> John :)
> 
> -Ursprüngliche Nachricht-
> Von: David Hubbard [mailto:[EMAIL PROTECTED] 
> Gesendet: Thursday, October 20, 2005 11:54 PM
> An: [EMAIL PROTECTED]
> Betreff: RE: Level3 problems
> 
> 
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> > 
> > On Fri, Oct 21, 2005 at 02:28:23AM -0400, Richard A 
> Steenbergen wrote:
> > > 
> > > On Fri, Oct 21, 2005 at 09:26:06AM +0300, Emilian Ursu wrote:
> > > > 
> > > > I see its completely down and several others are starting
> > > > to have problems.
> > > > Anyone knows whats up ?
> > > 
> > > I think everyone sees them completely down across the board 
> > (even mpls 
> > > transport services), been that way for about 30 mins now. :)
> > 
> > All of Speakeasy outbound SF (and possible other locations) is down,
> > after being on hold with them, they are saying Level3 and 
> not further
> > information yet.
> > 
> 
> I'm sure Cogent is getting a kick out of this. :-)
> Our L3 link started having serious issues at about 1:45 EST
> and we turned it off.  It seemed like an eternity for them
> to withdraw our routes from their peers though so they
> must be having some serious issues.  From home on Verizon
> FIOS they seem to be trying to push most of their traffic
> onto their Level 3 link in Atlanta which is making me
> unable to get to Google and a few other big sites.
> 
> For those with L3 that want to call them, they have parent
> ticket 1429209 open on this issue but you won't get any info
> yet so I'd give them some time.
> 
> Dave
> 
> 

RE: Level3 problems

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> 
> On Fri, Oct 21, 2005 at 02:28:23AM -0400, Richard A Steenbergen wrote:
> > 
> > On Fri, Oct 21, 2005 at 09:26:06AM +0300, Emilian Ursu wrote:
> > > 
> > > I see its completely down and several others are starting
> > > to have problems.
> > > Anyone knows whats up ?
> > 
> > I think everyone sees them completely down across the board 
> (even mpls 
> > transport services), been that way for about 30 mins now. :)
> 
> All of Speakeasy outbound SF (and possible other locations) is down,
> after being on hold with them, they are saying Level3 and not further
> information yet.
> 

I'm sure Cogent is getting a kick out of this. :-)
Our L3 link started having serious issues at about 1:45 EST
and we turned it off.  It seemed like an eternity for them
to withdraw our routes from their peers though so they
must be having some serious issues.  From home on Verizon
FIOS they seem to be trying to push most of their traffic
onto their Level 3 link in Atlanta which is making me
unable to get to Google and a few other big sites.

For those with L3 that want to call them, they have parent
ticket 1429209 open on this issue but you won't get any info
yet so I'd give them some time.

Dave

RE: Choosing new transit: software help?

[David Hubbard]

From: [EMAIL PROTECTED] 
> > 
> > I know most nanog responses seem to go off list immediately
> > but I'd be interested in this as well for traffic engineering.
> > A top AS and top prefix talkers would be really useful.
> 
>   perhaps you have forgotten this nifty set of pages:
> 
>   http://www.caida.org/analysis/topology/as_core_network/
> 

I meant top talkers as in top ones talking to my network.
I don't care about everyone else.  :-)

David

RE: Choosing new transit: software help?

[David Hubbard]

From: John Dupuy
> 
> We are looking at getting an additional transit connection.
> 
> In the past, we have used fixedorbit.com and the like and 
> "guesstimated" our best transit choices. (Other factors
> came into play as well, of course, such as price...)
> 
> Anyway, does anyone have a suggestion for determine our next 
> best transit?  Essentially, I am looking for techniques of:
> 
> 1. Gathering our current traffic patterns and subtotalling 
> source/destination IP by ASN.
> 2. Gathering our BGP views into a useful form for analysis.
> 3. Using #1 and #2 to analyze which new AS would make the 
> most sense to connect to for transit. The goal would be for
> the new transit to reduce the number of AS we must transit
> given our customer's actual usage.

I know most nanog responses seem to go off list immediately
but I'd be interested in this as well for traffic engineering.
A top AS and top prefix talkers would be really useful.

David

RE: Level 3's side of the story

[David Hubbard]

From: William Allen Simpson
> 
> I don't remember seeing this public notice from Level(3) posted
> Wouldn't that be "without notice from Level(3)"?

They notified Cogent, not the public.  Cogent chose to
not do anything other than hope they won the staring
contest when Level 3 terminated the link, which
apparently they did.  Be interesting to see what
will happen in a month when they go at it again.

> 
> Splendid, that gives the world sufficient time to accept 
> Cogent's offer of 1 year free service.

This is not the first time Cogent has used their customers
as pawns in peering disputes, I don't know if I'd jump on
the bandwagon so quickly (spoken as a customer of both
companies).

David

RE: Free service offer by Cogent

[David Hubbard]

From: Jeffrey Sharpe
> 
> Hello,
> 
> I was just wondering if anyone has taken Cogent up on the 
> offer for the Free year of service? Is it working good?
> Is it truly free? What is the catch? Or is it a publicity
> stunt by Cogent?
> 
> Anyone?
> 

I've got links to both providers; not sure if you'd consider
it a catch but I'd recommend getting the list of supported
BGP communities from Cogent before committing to them, you
might find you can't do things you're used to being able to.

David

Opinions on transit

[David Hubbard]

If anyone with a web hosting provider's perspective, or
even just an end user's perspective, could send me
off-list opinions on AT&T, MCI, XO, NTT and Sprint I'd
appreciate it.  Direct experience would be best, don't
really want hearsay.  I'm particularly interested in
how the various companies NOC's respond to less run of
the mill problems such as issues with other customers
of theirs having problems that affect you, BGP
communities and traffic engineering questions/answers
and then just plain performance, etc.

We've recently experienced problems where us and a mutual
customer of one of our transit providers couldn't talk
to each other due to a failing link on the other customer's
side.  That remote company is multi-homed so I just needed
to make them not prefer our common transit to get to us
side.  The transit provider does not allow me to prepend just
to that remote AS, accolades to providers that do offer
this, so our only option to resolve the problem was to
turn our link to this transit off, leave our customers
behind the remote ISP unable to get to us, or get the
transit provider to help.  Well, provider was useless
and told me to call their German NOC since the country
where the problem was is handled through there.  So,
we're going to replace them with one of the above and I
want some real world feedback to go along with the providers'
versions of how they perform and respond to support
requests.

Thanks,

David

PBR needing to hit the cpu?

[David Hubbard]

Just curious, do most vendors' hardware need to hit the
cpu when doing policy-based routing?  I found one of my
border routers' cpu's on the bad end of a DDoS but once
I turned off a not necessarily required setup to force
some outbound traffic to take a specific outbound link
via PBR, the DDoS traffic was no longer an issue.  It was
only about 200 Mbit so I hadn't expected it to be an issue
but apparently it was; I was surprised when support told
me the PBR was making traffic hit the cpu.  

TIA, 

David

RE: Calling all NANOG'ers - idea for national hardware price quote registry

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of [EMAIL PROTECTED]
> 
> Oh, and you can have up to 7 of  attached, unless you 
> have a  attached, in which case you can have 5 
> and one  or 4  and 2 .  You know the
> configurator drill.. ;)
> 
> Sure, go for it. :) 

Don't forget about "Well we're super sized network equipment
vendor X so if you rip out all of your equipment from vendor
Y, you can get discount + special discount."  :-)

David

RE: Technical contact at Cogent

[David Hubbard]

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> 
> Can someone from Cogent or with a technical contact there (other than 
> [EMAIL PROTECTED]) contact me offlist? I would like to discuss some BGP 
> issues with them. 
> 
> Thanks very much, 
> 
> Tao Wan
> [EMAIL PROTECTED]
> 

You've already tried calling them and gotten no response?
Unlike some other providers, Cogent's number on their web
site has always worked well for me.

David 

OT? Device to limit simultaneous connections per host?

[David Hubbard]

Hello everyone, I'm curious if anyone knows of a
device that can throttle or limit a remote
host's simultaneous connections or requests per
second for web traffic on a per-IP basis.  So I
don't want to say web server X can only have 100
simultaneous connections and 10 requests per second.
I want to say that for any given IP connecting to
web server X, any one IP can have no more than 5 open
connections and should be throttled if it starts
making more than ten requests per second.  If it
could even be url-aware in that it could only apply
the rules to specific types of web requests, that
would be even better.

The motivation here is to find a piece of equipment
that can protect compute-intensive, database-driven
websites from overly aggressive proxies, firewalls,
search engines, etc. which like to hammer a given
site with 50+ simultaneous requests against pages
that could potentially need a few seconds of
processing time per request.

I've looked at a Packeteer PacketShaper running
in reverse of what it normally would, trying to
throttle and shape requests against the server
rather than optimizing traffic for a low speed
link like it was designed, but that didn't really
work out as it could not have the policies applied
on a per remote IP basis.

Thanks,

David

Providers that support prepending to specific remote AS's?

[David Hubbard]

Hi all, I'd appreciate any on or offlist emails
with the names of larger providers that allow
you, through communities, to do prepending of
your AS path to selected remote AS's.  We use two
providers that allow this since we use the feature
but am wanting to dump one of our providers who
does not.

Basically we have a customer within AS X and us
and AS X both have transit through AS Y.  The link
between AS Y and AS X is seriously overloaded so our
customer is pretty much dead in the water since AS X
is a countrywide monopoly telco for the country in
question.  We've forced traffic to AS X to take
a different route in but since we can't path prepend
just to AS X through AS Y, we're stuck with the
only solution being disable the link to AS Y or
prepend to all of AS Y which we don't want to do.
AS Y doesn't feel the need to help since their view
is the customer of theirs has chosen to not upgrade
the oversubscribed link.

Thanks,

David

clec vs ilec, how do you know who's lying?

[David Hubbard]

Hello everyone, not sure if this is off topic or not
since it is will be operational in nature if I can ever
get the service set up. :-)  I'm having the pleasure, or
lack thereof, of ordering some data connectivity via a
very large clec which requires the ilec to provide the
local loops.  Well we're about two months past the
estimated install completion and all I get from the clec
is continuous blame pointed at the ilec who has now
missed three install dates and in turn has wasted staff
time sitting there from 8 to 5 each of the days; assuming
they were really scheduled in the first place.  I know the
two types of entities don't particularly like each other
but at this point how do I tell who's lying to me?  I
have supposed work order numbers for the ilec but I don't
have any direct contact with them to see if they are
real numbers and if the disposition of the previous
work orders are what the clec has told me or if they are
messing things up themselves and trying to cover it up.

Thanks,

David

SBC (AS 7132) contact?

[David Hubbard]

Seeing some random issues with hosting customers
of ours unable to get to their sites.  Traces out
of 7132 seem to stop in and around some of the
following:

core2-p10-0.crrvca.sbcglobal.net (151.164.42.5)
core1-p1-0.crrvca.sbcglobal.net (151.164.188.45)
core2-p3-0.cranca.sbcglobal.net (151.164.243.233)
core1-p1-0.cranca.sbcglobal.net (151.164.241.221)
core2-p11-0.crscca.sbcglobal.net (151.164.242.81)

Support from them to our mutual customers won't
go past "try clearing your cache" so I'd like to
find someone in net ops that can help. :-)

Thanks,

David

RE: Load Testing Util

[David Hubbard]

Iperf works really well:

http://dast.nlanr.net/Projects/Iperf/ 

It will do tcp/udp/multicast; you can pick the rate on
the udp side and the client/server architecture lets you
measure jitter, out of order packets, loss, etc.
So you could load up a fixed rate of udp and then produce
your burst using a second copy, etc. to see what happens.

David

> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 
> Behalf Of Dan Mahoney, System Admin
> Sent: Friday, June 03, 2005 1:49 PM
> To: nanog@merit.edu
> Subject: Load Testing Util
> 
> 
> Hey all,
> 
> Does anyone know of a (preferably opensource) tool that can generate 
> network loads of specific protocols and/or levels (for example, if I 
> wanted to see how much loss I got on a 1 meg spike, over time).  I'm 
> hopefully looking for something client/server so I'm not necessarily 
> dependent on reply packets to measure the thoroughput.
> 
> If anyone has any ideas, I'd love to hear them.  Privately if 
> you like, as 
> I realize the potential for misuse of such things.
> 
> -Dan
> 
> --
> 
> Dan Mahoney
> Techie,  Sysadmin,  WebGeek
> Gushi on efnet/undernet IRC
> ICQ: 13735144   AIM: LarpGM
> Site:  http://www.gushi.org
> ---
> 
> 
> 

RE: DNS requests and Bandwidth

[David Hubbard]

From: aljuhani
> 
>  
> One way was to block requests from that DNS IP but that was not
> practicle as many users on that DNS won't be able to communicate
> with our server.
>  
> so What is the best way to prevent DNS queries consuming bandwidth.

Run an authoritative-only DNS server that won't respond
to queries for domains it doesn't handle.  tinydns from
http://cr.yp.to/djbdns.html would be an example.

David

Contact for blacklist.sequoia.ops.asp.att.net RBL?

[David Hubbard]

Anyone know who at att.net runs the
blacklist.sequoia.ops.asp.att.net RBL?  When they block
based on it, they tell the sender to contact their ISP
and give no method of contact for themselves.

Thanks,

David

RE: Ironcore foundry

[David Hubbard]

Issam Hakimi [ Killix ] wrote:
> I am in the search of documentation on the ironcore generation of the
> routers foundry. All the urls are the welcomes.
> Thanks.
> 
> Regards,
> Issam Hakimi

http://www.foundrynet.com/services/documentation/index.html

David

Resolution -> RE: Known communities for AS174?

[David Hubbard]

Talked with Cogent IP Engineering today, was doing my own
prepending in the meantime.  I received a number of replies
on and off list with quite a bit of conflicting info from
Cogent doesn't support any communities other than do not
announce to they support this or that to references of RIPE
and RADb both of whom seem to have some outdated info.
It turns out they do support a bunch of communities,
including path prepending of one, two and three times which
are what I was looking for, and for at least the US market,
they're all in the customer user guide which I hadn't
received; no I really didn't receive it not just didn't
read it. :-)

The person I spoke with was going to have someone update
RADb with current information as what's there is about
eight months old I think.  I ended up having to clear
the peer session hard but then my community was applied
and is functioning as intended.  Only thing missing that
would be really nice would be something like Level 3
where you can selectively prepend to a specific peer AS
of theirs. :-)  But they were very responsive as soon as
got in touch with the right group.

David

Known communities for AS174?

[David Hubbard]

Hello, I'm looking for a way to do path prepending
for my prefix as it leaves AS174 (Cogent), one of my
upstreams.  The following:

http://www.radb.net/cgi-bin/radb/whois.cgi?obj=AS174

suggests that at least as recently as last May they
might have accepted:

  3. Communities controlling Cogents AS path prepending
 for customer routes on egress:

 community   effect
 174:3000   do not announce
 174:3001   prepend 174 1 time
 174:3002   prepend 174 2 times
 174:3003   prepend 174 3 times

But I've tried setting each of those and it doesn't
seem to have any effect.  Anyone know if that info is
out of date or maybe has something else to try?

Thanks,

David

Bank One 159.53.0.0/16 contact?

[David Hubbard]

Anyone have a clueful contact at Bank One?  Their
ARIN POC info is some generic switchboard that is
completely unrelated to their allocation and who
refuses to connect you to anyone in datacomm if
you don't know a specific contact name to ask for.
They told me that they'd be happy to write down
what was supposedly being attacked by Bank One's
network, although they found that hard to believe,
and get me a response from a manager if one felt
it was appropriate though but that may take a day
or two.  Nice...

Thanks,

David