RE: DNS issues?

2007-07-20 Thread Jon Lewis


On Fri, 20 Jul 2007, Erik Amundson wrote:


I was having issues reaching any of the roots, and any of the TLDs for
.com and .net.  I tested them directly, and had no luck.


It's more likely they were having trouble reaching you.  My guess is that 
much of your return traffic normally comes to you over your XO connection. 
During whatever routing issue XO had, they continued to propogate your 
route(s) and receive traffic for you from the "internet", but were unable 
to route that traffic across their network to you.


I've seen similar things happen with other providers.  During such 
breakage, shutting your BGP session to them doesn't always help.



I was merely looking to see if anything was broken elsewhere in the
world to see if it was just my network so I could diagnose things a bit
better.


With the distributed nature of the DNS roots, if all the root servers 
suddenly "break", you can be nearly 100% sure the problem is much closer 
to the observer than to all the root servers or the "internet" in general.


--
 Jon Lewis   |  I route
 Senior Network Engineer |  therefore you are
 Atlantic Net|
_ http://www.lewis.org/~jlewis/pgp for PGP public key_


Re: DNS issues?

2007-07-20 Thread Randy Bush

Erik Amundson wrote:
> I was having issues reaching any of the roots, and any of the TLDs for
> .com and .net.  I tested them directly, and had no luck.

how did you test?  please cut and paste.  "it did not work," is pretty
useless.

randy



RE: DNS issues?

2007-07-20 Thread Erik Amundson

I was having issues reaching any of the roots, and any of the TLDs for
.com and .net.  I tested them directly, and had no luck.

I was merely looking to see if anything was broken elsewhere in the
world to see if it was just my network so I could diagnose things a bit
better.

XO's problem eventually got fixed (relatively), and I was suddenly able
to get answers from the roots/TLDs again over Verizon
Business/UUNET/MCI/WORLDCOM/whatever, and Qwest after about 45 minutes.
I have found no explanation for the loss of connectivity as of yet.

I found it strange that I had issues reaching the name serves even after
shutting off XO, so that's why I asked...

Some of NANOG folks have been helpful and kind, some not, but I read
NANOG a lot, and I'm not surprised. :)

ThanksI guess.


Erik Amundson


Re: DNS issues?

2007-07-20 Thread Randy Bush

Buhrmaster, Gary wrote:
> By answering, we are enablers.

it is not possible to answer as there was no actual answerable question,
so this is not an issue

> By not answering, we are unhelpful elitists.

i have no problem with whatever fools wish to call me.  i care for the
opinions of the wise.

randy


RE: DNS issues?

2007-07-20 Thread Buhrmaster, Gary

By answering, we are enablers.
By not answering, we are unhelpful elitists.

Rock, meet hard place.

> we will probably never know.  folk who say "the internet broke" seem
> unable to actually waste their time giving us actual technical
> information.  this seems matched to a willingness to waste our time
> guessing.


Re: DNS issues?

2007-07-20 Thread Randy Bush

[EMAIL PROTECTED] wrote:
> "Tune in tomorrow for updates".  Most likely, it's a TLD server or
> three that he's unable to reach, not the actual roots themselves.

we will probably never know.  folk who say "the internet broke" seem
unable to actually waste their time giving us actual technical
information.  this seems matched to a willingness to waste our time
guessing.

randy


Re: DNS issues?

2007-07-20 Thread Valdis . Kletnieks
On Fri, 20 Jul 2007 19:52:34 -, =?utf-8?B?Q2hyaXN0aWFuIEt1aHR6?= said:

> To Vladis' point, how do you know that you couldn't reach the roots vs the
> roots not being able to reach you?

In addition to which, his nameservers probably wouldn't *need* to reach the
actual roots unless they'd managed to timeout their cache for things like
.com and .org and .net.  'dig . soa' says:

.   86400   IN  SOA A.ROOT-SERVERS.NET. 
NSTLD.VERISIGN-GRS.COM. 2007072000 1800 900 604800 86400

86400. Yummy. "Tune in tomorrow for updates".  Most likely, it's a TLD server
or three that he's unable to reach, not the actual roots themselves. But given
that routing issues *are* included in the original message, and I've seen naught
saying that there were any DNS servers in the "reachable but misbehaving" state,
I have to conclude that root cause is a routing blurp.


pgpEhKNccv1q7.pgp
Description: PGP signature


Re: DNS issues?

2007-07-20 Thread Christian Kuhtz
To Vladis' point, how do you know that you couldn't reach the roots vs the 
roots not being able to reach you?

--
Sent from my BlackBerry.  

-Original Message-
From: "Erik Amundson" <[EMAIL PROTECTED]>

Date: Fri, 20 Jul 2007 14:39:37 
Cc:
Subject: RE: DNS issues?



I also have UUNET and Qwest routes working fine, but even on those, I couldn't 
reach the roots for quite a while.

Erik Amundson
IT Infrastructure Manager
Open Access Technology Int'l, Inc.
Phone (763) 201-2005
mailto:[EMAIL PROTECTED]
 
CONFIDENTIAL INFORMATION:  This email and any attachment(s) contain 
confidential and/or proprietary information of Open Access Technology 
International, Inc.  Do not copy or distribute without the prior written 
consent of OATI.  If you are not a named recipient to the message, please 
notify the sender immediately and do not retain the message in any form, 
printed or electronic.


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] 
Sent: Friday, July 20, 2007 2:34 PM
To: Erik Amundson
Cc: nanog@nanog.org
Subject: Re: DNS issues?

On Fri, 20 Jul 2007 14:25:41 CDT, Erik Amundson said:

> We just lost DNS totally for a while and just got it back...XO 
> communications also lost almost all Internet routes at the same time...

Cause (losing all routes) -> effect (can't reach stuff other side of the 
routes, including DNS servers)?

Sounds like an XO routing issue, not a DNS issue.


Re: DNS issues?

2007-07-20 Thread Randy Bush

Erik Amundson wrote:
> I also have UUNET and Qwest routes working fine, but even on those, I
> couldn't reach the roots for quite a while.

actual technical description of symptoms would be helpful.  e.g. could
not ping/trace to some list of root server addresses?

randy


Re: Anyone seeing Level3 issues in Dallas ?

2007-07-20 Thread Jason Seemann

Our representative told us they had a massive fiber cut in the Dallas metro
area that affected a number of connections.

Jason


On 7/20/07, W. Kevin Hunt <[EMAIL PROTECTED]> wrote:



Yeah, this morning Broadwing/L3 had yet another failure, and it took them
hours to figure out what the hell was wrong.


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Scott Francis
Sent: Friday, July 20, 2007 11:39 AM
To: W. Kevin Hunt
Cc: nanog@nanog.org
Subject: Re: Anyone seeing Level3 issues in Dallas ?


panther reports issues between L3 and e.g. Savvis, GBLX, PCCW and
others in the LAX/SFO area (we're having performance issues with some
of their CDN nodes in those areas right now). Possibly more widespread
than just California ...
--
[EMAIL PROTECTED],darkuncle.net} || 0x5537F527
encrypted email to the latter address please
http://darkuncle.net/pubkey.asc for public key

On 7/19/07, W. Kevin Hunt <[EMAIL PROTECTED]> wrote:
>
> All my traces die one hop into Broadwing/L3 Dallas.
> After noticing it, I killed my peer w/ them, and still had issues
> getting to anyone I know prefers L3 for peering.
> I turned it back up when I saw some L3 destinations working via TW
[snip]




RE: DNS issues?

2007-07-20 Thread Erik Amundson

I also have UUNET and Qwest routes working fine, but even on those, I couldn't 
reach the roots for quite a while.

Erik Amundson
IT Infrastructure Manager
Open Access Technology Int'l, Inc.
Phone (763) 201-2005
mailto:[EMAIL PROTECTED]
 
CONFIDENTIAL INFORMATION:  This email and any attachment(s) contain 
confidential and/or proprietary information of Open Access Technology 
International, Inc.  Do not copy or distribute without the prior written 
consent of OATI.  If you are not a named recipient to the message, please 
notify the sender immediately and do not retain the message in any form, 
printed or electronic.


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] 
Sent: Friday, July 20, 2007 2:34 PM
To: Erik Amundson
Cc: nanog@nanog.org
Subject: Re: DNS issues?

On Fri, 20 Jul 2007 14:25:41 CDT, Erik Amundson said:

> We just lost DNS totally for a while and just got it back...XO 
> communications also lost almost all Internet routes at the same time...

Cause (losing all routes) -> effect (can't reach stuff other side of the 
routes, including DNS servers)?

Sounds like an XO routing issue, not a DNS issue.


Re: DNS issues?

2007-07-20 Thread Valdis . Kletnieks
On Fri, 20 Jul 2007 14:25:41 CDT, Erik Amundson said:

> We just lost DNS totally for a while and just got it back...XO
> communications also lost almost all Internet routes at the same time...

Cause (losing all routes) -> effect (can't reach stuff other side of the routes,
including DNS servers)?

Sounds like an XO routing issue, not a DNS issue.


pgpbqQmCwAmV9.pgp
Description: PGP signature


DNS issues?

2007-07-20 Thread Erik Amundson
We just lost DNS totally for a while and just got it back...XO
communications also lost almost all Internet routes at the same time...

 

Anyone else noticing weirdness?

 

 

Erik Amundson



 



RE: Anyone seeing Level3 issues in Dallas ?

2007-07-20 Thread W. Kevin Hunt

Yeah, this morning Broadwing/L3 had yet another failure, and it took them
hours to figure out what the hell was wrong.


-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Scott Francis
Sent: Friday, July 20, 2007 11:39 AM
To: W. Kevin Hunt
Cc: nanog@nanog.org
Subject: Re: Anyone seeing Level3 issues in Dallas ?


panther reports issues between L3 and e.g. Savvis, GBLX, PCCW and
others in the LAX/SFO area (we're having performance issues with some
of their CDN nodes in those areas right now). Possibly more widespread
than just California ...
-- 
[EMAIL PROTECTED],darkuncle.net} || 0x5537F527
encrypted email to the latter address please
http://darkuncle.net/pubkey.asc for public key

On 7/19/07, W. Kevin Hunt <[EMAIL PROTECTED]> wrote:
>
> All my traces die one hop into Broadwing/L3 Dallas.
> After noticing it, I killed my peer w/ them, and still had issues
> getting to anyone I know prefers L3 for peering.
> I turned it back up when I saw some L3 destinations working via TW
[snip]



Weekly Routing Table Report

2007-07-20 Thread Routing Analysis Role Account

This is an automated weekly mailing describing the state of the Internet
Routing Table as seen from APNIC's router in Japan.
Daily listings are sent to [EMAIL PROTECTED]

For historical data, please see http://thyme.apnic.net.

If you have any comments please contact Philip Smith <[EMAIL PROTECTED]>.

Routing Table Report   04:00 +10GMT Sat 21 Jul, 2007

Report Website: http://thyme.apnic.net
This report:http://thyme.apnic.net/ap-data/2007/07/21/0400

Analysis Summary


BGP routing table entries examined:  226170
Prefixes after maximum aggregation:  118939
Deaggregation factor:  1.90
Unique aggregates announced to Internet: 109985
Total ASes present in the Internet Routing Table: 25742
Prefixes per ASN:  8.79
Origin-only ASes present in the Internet Routing Table:   22440
Origin ASes announcing only one prefix:   10932
Transit ASes present in the Internet Routing Table:3302
Transit-only ASes present in the Internet Routing Table: 74
Average AS path length visible in the Internet Routing Table:   3.6
Max AS path length visible:  22
Max AS path prepend of ASN (35389)   16
Prefixes from unregistered ASNs in the Routing Table: 9
Unregistered ASNs in the Routing Table:  11
Prefixes from 32-bit ASNs in the Routing Table:   4
Special use prefixes present in the Routing Table:0
Prefixes being announced from unallocated address space: 11
Number of addresses announced to Internet:   1739111776
Equivalent to 103 /8s, 168 /16s and 189 /24s
Percentage of available address space announced:   46.9
Percentage of allocated address space announced:   62.2
Percentage of available address space allocated:   75.5
Total number of prefixes smaller than registry allocations:  119103

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:53054
Total APNIC prefixes after maximum aggregation:   20706
APNIC Deaggregation factor:2.56
Prefixes being announced from the APNIC address blocks:   49994
Unique aggregates announced from the APNIC address blocks:21984
APNIC Region origin ASes present in the Internet Routing Table:3017
APNIC Prefixes per ASN:   16.57
APNIC Region origin ASes announcing only one prefix:804
APNIC Region transit ASes present in the Internet Routing Table:459
Average APNIC Region AS path length visible:3.7
Max APNIC Region AS path length visible: 22
Number of APNIC addresses announced to Internet:  304990112
Equivalent to 18 /8s, 45 /16s and 199 /24s
Percentage of available APNIC address space announced: 75.5

APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911
APNIC Address Blocks   58/7, 60/7, 116/6, 120/6, 124/7, 126/8, 202/7
   210/7, 218/7, 220/7 and 222/8

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:107407
Total ARIN prefixes after maximum aggregation:62361
ARIN Deaggregation factor: 1.72
Prefixes being announced from the ARIN address blocks:79061
Unique aggregates announced from the ARIN address blocks: 31348
ARIN Region origin ASes present in the Internet Routing Table:11687
ARIN Prefixes per ASN: 6.76
ARIN Region origin ASes announcing only one prefix:4510
ARIN Region transit ASes present in the Internet Routing Table:1070
Average ARIN Region AS path length visible: 3.4
Max ARIN Region AS path length visible:  16
Number of ARIN addresses announced to Internet:   339579648
Equivalent to 20 /8s, 61 /16s and 147 /24s
Percentage of available ARIN address space announced:  75.0

ARIN AS Blocks 1-1876, 1902-2042, 2044-2046, 2048-2106
(pre-ERX allocations)  2138-2584, 2615-2772, 2823-2829, 2880-3153
   3354-4607, 4865-5119, 5632-6655, 6912-7466
   7723-8191, 10240-12287, 13312-15359, 16384-17407
   18432-20479, 21504-23551, 25600-26591,
   26624-27647, 29696-30719, 31744-33791
   35840-36863, 39936-40959
ARIN Address Blocks24/8, 63

Re: Anyone seeing Level3 issues in Dallas ?

2007-07-20 Thread Scott Francis


On 7/20/07, Scott Francis <[EMAIL PROTECTED]> wrote:

panther reports issues between L3 and e.g. Savvis, GBLX, PCCW and
others in the LAX/SFO area (we're having performance issues with some
of their CDN nodes in those areas right now). Possibly more widespread
than just California ...


master ticket for PHX metro: 61138

other areas may be affected as well; this is the only master ticket
number I've got.
--
[EMAIL PROTECTED],darkuncle.net} || 0x5537F527
   encrypted email to the latter address please
   http://darkuncle.net/pubkey.asc for public key


Re: Anyone seeing Level3 issues in Dallas ?

2007-07-20 Thread Scott Francis


panther reports issues between L3 and e.g. Savvis, GBLX, PCCW and
others in the LAX/SFO area (we're having performance issues with some
of their CDN nodes in those areas right now). Possibly more widespread
than just California ...
--
[EMAIL PROTECTED],darkuncle.net} || 0x5537F527
   encrypted email to the latter address please
   http://darkuncle.net/pubkey.asc for public key

On 7/19/07, W. Kevin Hunt <[EMAIL PROTECTED]> wrote:


All my traces die one hop into Broadwing/L3 Dallas.
After noticing it, I killed my peer w/ them, and still had issues
getting to anyone I know prefers L3 for peering.
I turned it back up when I saw some L3 destinations working via TW

[snip]