Re: Problems with NS*.worldnic.com
- Original Message - From: "Randy Bush" <[EMAIL PROTECTED]> To: "Christopher L. Morrow" <[EMAIL PROTECTED]> Cc: Sent: Tuesday, April 26, 2005 16:35 Subject: Re: Problems with NS*.worldnic.com > > lots of folk sent email to me and not the list. most report > worldnic responding with tcp 53 and not udp. would love to > hear confirmation on list. can think of a number of causes, > one possible, but just a stab in the dark, would be an > intentional hack as a defense to a spoofed-ip attack. That is a bind issue when receiving empty response from worldnic ns on udp queries, it asks again on tcp which is very slow. more here: http://isc.sans.org/diary.php?date=2005-04-22 > what are some names known to be hosted on worldnic? > > randy > aljuhani
Re: Problems with NS*.worldnic.com
In message <[EMAIL PROTECTED]>, "Christ opher L. Morrow" writes: > > >On Tue, 26 Apr 2005, Randy Bush wrote: > >> lots of folk sent email to me and not the list. most report >> worldnic responding with tcp 53 and not udp. would love to >> hear confirmation on list. can think of a number of causes, >> one possible, but just a stab in the dark, would be an >> intentional hack as a defense to a spoofed-ip attack. >> >> what are some names known to be hosted on worldnic? > >we had problems reported with: > >www.calairmail.com >www.holidaycardwebsite.com > >I did some poking around lastnight with dig and some local unix hosts that >I hadn't tried this before on and got no change to tcp :( (so no truncate >and returned results via UDP) though today I see: > >[EMAIL PROTECTED]:~$ dig www.holidaycardwebsite.com. @ns7.worldnic.com >;; Truncated, retrying in TCP mode. > >and failures (which is PROBABLY my silly iptables config...) > >[EMAIL PROTECTED]:~$ dig www.holidaycardwebsite.com. @ns8.worldnic.com > >; <<>> DiG 9.2.2rc1 <<>> www.holidaycardwebsite.com. @ns8.worldnic.com >;; global options: printcmd > >interesting that both servers aren't doing the same thing? > Both work for me, from two different places, one of which has v6 connectivity and one of which doesn't. --Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Re: Problems with NS*.worldnic.com
On Tue, 26 Apr 2005, Brett Frankenberger wrote: > On Tue, Apr 26, 2005 at 01:22:41PM +, Christopher L. Morrow wrote: > > > > On Tue, 26 Apr 2005, Simon Waters wrote: > > > > > The worldnic.com and worldnic.net appear to use the MMDDVV convention > > > for > > > SOA serial numbers, and so it would appear nothing has changed their end > > > in > > > > Interesting, I thought the worldnic.com servers were NSI's 'free hosting > > for domains you registered through us' servers, which would imply they get > > changed 'frequently' no? > > I think he's talking about the worldnic.com and worldnic.net zones > themselves. Those wouldn't need to change unless new names were added yup, this was clarified off-list :( I'll take my pre-coffee lumps on that one :( where is that coffee pot?
Re: Problems with NS*.worldnic.com
On Tue, 26 Apr 2005, Randy Bush wrote: > lots of folk sent email to me and not the list. most report > worldnic responding with tcp 53 and not udp. would love to > hear confirmation on list. can think of a number of causes, > one possible, but just a stab in the dark, would be an > intentional hack as a defense to a spoofed-ip attack. > > what are some names known to be hosted on worldnic? we had problems reported with: www.calairmail.com www.holidaycardwebsite.com I did some poking around lastnight with dig and some local unix hosts that I hadn't tried this before on and got no change to tcp :( (so no truncate and returned results via UDP) though today I see: [EMAIL PROTECTED]:~$ dig www.holidaycardwebsite.com. @ns7.worldnic.com ;; Truncated, retrying in TCP mode. and failures (which is PROBABLY my silly iptables config...) [EMAIL PROTECTED]:~$ dig www.holidaycardwebsite.com. @ns8.worldnic.com ; <<>> DiG 9.2.2rc1 <<>> www.holidaycardwebsite.com. @ns8.worldnic.com ;; global options: printcmd interesting that both servers aren't doing the same thing?
Re: Problems with NS*.worldnic.com
Randy Bush <[EMAIL PROTECTED]> wrote: > lots of folk sent email to me and not the list. most report worldnic > responding with tcp 53 and not udp. would love to hear confirmation > on list. can think of a number of causes, one possible, but just a > stab in the dark, would be an intentional hack as a defense to a > spoofed-ip attack. That's quite an interesting theory, and you may be right. However, when given the choice between incompetence and malice, I know which one my money is on. > what are some names known to be hosted on worldnic? voipbuster.com's one that they've been whining about on uk.telecom. Right now, UDP DNS requests to ns25/ns26.worldnic.com for that domain are giving truncated responses and TCP calls aren't even being answered, so it's even more buggered than the last time I poked at it. -- "I Adjure Thee, O Foul Demon of The Sinus, by this Leatherman Tool and this Fully Earthed 30 Amp Power Strip! Remain Thou within the Faraday Cage and Answer the Questions put to Thee, and I shall Discharge Thee that Thou mayest return from Whence Thou Camest." -- Peter da Silva
Re: Problems with NS*.worldnic.com
At 21:34 -0700 4/25/05, Rodney Joffe wrote: The culprit is dig. Ahh, dig. What version? You have to be running the latest at all times these days...so many changes... In my experiences with v6 the problems I have come down two are: 1) Broken testing tools. (See change 1610 in the BIND CHANGES file for one.) 2) Broken route policy. (Dasterdly ISP's!) 3) Broken OS API's. (Have we learned nothing since or from Berkeley Sockets?) #1 - I've had to reevaluate everything I know about debugging since I met IPv6. Now there's an entirely alternate universe of failure to consider. One day I was sitting in RIPE NCC's offices and couldn't 'dig @ns.ripe.net'. So I walked to the ops room and asked, "umm, is your big machine down." After a good laugh, we figured that my Mac was trying v6 where v6 wasn't *really* live. #2 - When I first got real live IPv6 service from a provider, I tried tracerouting to all the machines I knew about - the roots as listed on root-servers.org, the RIPE machines. I'd get about halfway there and fail. I asked for reverse traces from the other side and see failures about the same place. We had to work with ISPs to loosen route policies. #3 - I have seen all sorts of mistakes involving OS's, OS API's, and app software API's. Mapped addresses are mishandled, having more than one address to try is something apps don't deal with. (Like they've been force fed one kind of food their entire life, and now have to choose from a menu.) At NANOG last year I related my problems with ssh (choosing v6 over v4 - and me assigning the same domain name to two machines, one on a v4 net and one on a v6 net). Stupid me... The biggest problem was that one type of machine kept dropping its statically configured default v6 route. Packets would get in, but they didn't know where to go next. The machine logged all activity as good though...it didn't know. -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Edward Lewis+1-571-434-5468 NeuStar If you knew what I was thinking, you'd understand what I was saying.
Re: Problems with NS*.worldnic.com
lots of folk sent email to me and not the list. most report worldnic responding with tcp 53 and not udp. would love to hear confirmation on list. can think of a number of causes, one possible, but just a stab in the dark, would be an intentional hack as a defense to a spoofed-ip attack. what are some names known to be hosted on worldnic? randy
Re: Problems with NS*.worldnic.com
On Tue, 26 Apr 2005, Simon Waters wrote: > Have to say we see no issues here with the worldnic.com nameservers, other > than they appear to be located on the same physical network. > > I think people should post queries that fail, including date/time, and full > "dig" output for that query from the server they used, and the version of > recursive nameserver used. Otherwise it is purely speculative guess work to > figure out if it is a DNS delegation issue, or something else (network > congestion?). I think I suggested similar yesterday as did Mr. Bush. > The worldnic.com and worldnic.net appear to use the MMDDVV convention for > SOA serial numbers, and so it would appear nothing has changed their end in > terms of zone data for at least five months in terms of zone file settings. Interesting, I thought the worldnic.com servers were NSI's 'free hosting for domains you registered through us' servers, which would imply they get changed 'frequently' no?
Re: Problems with NS*.worldnic.com
Suresh Ramasubramanian wrote: I'd say fix the resolver to not try resolve v6 where there exists no v6 connectivity I'd say fix the broken v6 connectivity. - Kevin
Re: Problems with NS*.worldnic.com
Have to say we see no issues here with the worldnic.com nameservers, other than they appear to be located on the same physical network. I think people should post queries that fail, including date/time, and full "dig" output for that query from the server they used, and the version of recursive nameserver used. Otherwise it is purely speculative guess work to figure out if it is a DNS delegation issue, or something else (network congestion?). No one should be surprised that a DNS request may be truncated and switched to TCP, that is in the RFCs. Although the servers in question run BIND9 so presumably support EDNS0, which suggests those seeing truncation may be running rather old code, or unusual recursive resolvers. The worldnic.com and worldnic.net appear to use the MMDDVV convention for SOA serial numbers, and so it would appear nothing has changed their end in terms of zone data for at least five months in terms of zone file settings. All looks rosy from here.
Re: Problems with NS*.worldnic.com
On Mon, 25 Apr 2005 22:19:51 PDT, "william(at)elan.net" said: > Perhaps a solution is to specifically enable ipv6 dns resolution as > preferable to ipv4 or the other way around. This could perhaps be > switch in resolv.conf or nsswitch.conf. Something like: > /etc/resolv.conf > search example.com > protocol ipv6 ipv4 At least on my system, there's an 'options inet6' line that makes it look for records, and mapping ipv4 into ipv6 addresses if only an A record is found. Also note that it doesn't fix the problem that's being seen - I might be able to contact the nameservers listed in resolv.conf via both IPv4 and IPv6 - the fun starts when my nameserver gets an NS entry that contains an record, and the nameserver has enough IPv6 connectivity to think it's worth a try, but you can't get there from here... pgpnXnSYEg9RG.pgp Description: PGP signature
Re: Problems with NS*.worldnic.com
something *very* strange is going on. the worldnic servers have been giving delayed or no results for days now. and nsi is hoping we and the wsj/nyt won't notice. I agree 100%. but it's probably time for us all to dump symptoms here and figure it out as a community, as the dog with the bone ain't 'fessing up. randy I'll bite. I couldn't resolve ns*.worldnic.com domains until I finally bit the bullet, and unblocked port 53 TCP from my DNS server. Then it worked fine. (after a few tries) I'm using BIND 9.2.4 without the eye pee vee six stuff compiled in. Because I don't want to start something; No discussion about me blocking port 53, ok? I got tired of gobs of log files of script kiddies trying to download my domains 5 years ago... I actually READ my logs besides, I had to keep the linux boxes safe from the tyranny of bind 8 until they got upgraded. :-) -Jerry
Re: Problems with NS*.worldnic.com
On Tue, 26 Apr 2005 [EMAIL PROTECTED] wrote: On Mon, 25 Apr 2005 21:34:54 PDT, Rodney Joffe said: I am not sure whether the correct solution is to "fix" dig so that is tries ipv4, or to get the os "fixed" on a dual stack capable system so that if there is not ipv6 connectivity it disables that part of the system. I suspect the first is appropriate, because there are obviously internal processes that may validly want to use ipv6 even though there is no ipv6 connection. The problem is that you *could* have local/campuswide ipv6 connectivity, but not have an IPv6 connection to the outside world. So my system comes up, it sees a Router Advertisement, it can get to other IPv6 systems that are 3-4 hops away. So how is it supposed to "know" that it doesn't have an ipv6 connection? Presumably the same way it "knows" it doesn't have an ipv4 connection when your OC-moby to the outside world falls over, but it's still perfectly able to talk to the entire rest of your corporate network Perhaps a solution is to specifically enable ipv6 dns resolution as preferable to ipv4 or the other way around. This could perhaps be switch in resolv.conf or nsswitch.conf. Something like: /etc/nsswitch.conf ... hosts: files dns dns-resolver: [NOTFOUND=return] A6 A Note: in this meaning NOTFOUND is only true when NXDOMAIN but not for NODATA OR /etc/resolv.conf search example.com protocol ipv6 ipv4 -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: Problems with NS*.worldnic.com
> So how is it supposed to "know" that it doesn't have an ipv6 connection? in my case, because o no interfaces have v6 addresses o v6 stack is not present o ... it should also not use smoke signals, analog voice phone, ... the chances of a box having a v6 connection to *anything* today is low, and should not be a reason to *break* v4 services. randy
Re: Problems with NS*.worldnic.com
On Mon, 25 Apr 2005 21:34:54 PDT, Rodney Joffe said: > I am not sure whether the correct solution is to "fix" dig so that is tries > ipv4, or to get the os "fixed" on a dual stack capable system so that if > there is not ipv6 connectivity it disables that part of the system. I > suspect the first is appropriate, because there are obviously internal > processes that may validly want to use ipv6 even though there is no ipv6 > connection. The problem is that you *could* have local/campuswide ipv6 connectivity, but not have an IPv6 connection to the outside world. So my system comes up, it sees a Router Advertisement, it can get to other IPv6 systems that are 3-4 hops away. So how is it supposed to "know" that it doesn't have an ipv6 connection? Presumably the same way it "knows" it doesn't have an ipv4 connection when your OC-moby to the outside world falls over, but it's still perfectly able to talk to the entire rest of your corporate network pgpAu9ps6nKO8.pgp Description: PGP signature
Re: Problems with NS*.worldnic.com
On 4/26/05, Rodney Joffe <[EMAIL PROTECTED]> wrote: > > The culprit is dig. > > I am not sure whether the correct solution is to "fix" dig so that is tries > ipv4, or to get the os "fixed" on a dual stack capable system so that if > there is not ipv6 connectivity it disables that part of the system. I I'd say fix the resolver to not try resolve v6 where there exists no v6 connectivity -srs
Re: Problems with NS*.worldnic.com
Randy, and others with this issue... On 4/25/05 5:24 PM, "Randy Bush" <[EMAIL PROTECTED]> wrote: > > something *very* strange is going on. the worldnic servers have > been giving delayed or no results for days now. and nsi is hoping > we and the wsj/nyt won't notice. > > i don't think this > > roam.psg.com:/usr/home/randy> doc -p -w worldnic.net > Doc-2.1.4: doc -p -w worldnic.net > Doc-2.1.4: Starting test of worldnic.net. parent is net. > Doc-2.1.4: Test date - Mon Apr 25 14:20:45 HST 2005 > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @a.gtld-servers.net. for NS of worldnic.net. failed > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @b.gtld-servers.net. for NS of worldnic.net. failed > > is the worldnic problem, but could be. but it is a problem. (i > generally ignore b root issues). > > but it's probably time for us all to dump symptoms here and figure > it out as a community, as the dog with the bone ain't 'fessing up. I spent some time two months ago chasing this down with the same two gtld-servers.net records, on my mac. The culprit is dig. On any system that is both ipv4 and ipv6 enabled, *even if there is no ipv6 connectivity*, dig - which has no awareness of ipv6 v. ipv4 - attempts to connect using the ipv6 address if both an ipv6 and an ipv4 address is provided. When it fails to connect, it fails. It does not do the "better" thing, which is to try another address for the same hostname, in this case the ipv4 address. If you attempt a dig for the ipv4 address of a.gtld-servers.net or b.gtld-servers.net, you will get your answer. I am not sure whether the correct solution is to "fix" dig so that is tries ipv4, or to get the os "fixed" on a dual stack capable system so that if there is not ipv6 connectivity it disables that part of the system. I suspect the first is appropriate, because there are obviously internal processes that may validly want to use ipv6 even though there is no ipv6 connection. I also suspect that the same thing would occur for an A record that had multiple ipv4 addresses in a round robin configuration, but where the "first" ip address was unreachable, the behavior would be the same as if it was an ipv6 address being tried on a system that had no ipv6 connectivity. But I have not taken the time to test. I did ask the maintainers of dig to look at this, and perhaps provide a solution. Rodney Joffe CenterGate Research Group, LLC http://www.centergate.com "Technology so advanced, even WE don't understand it"(R)
Re: Problems with NS*.worldnic.com
Matt Larson wrote: a.gtld-servers.net and b.gtld-servers.net have records. Some applications and stacks try the v6 address first if it's available and will appear to hang if you don't have v6 connectivity. That may very well be what's happening here. Are the records for a & b.gtld-servers.net new?
Re: Problems with NS*.worldnic.com
Well, the first thing any engineer worth their saly would ask in a situatin such as this is "Were any changes implemented, concurrent with the appearance of these problems, which would have possibly account for this?" This problem has fairly wide-spread implications, it would appear, and the lack of technical information on it is rather alarming... - ferg -- Matt Larson <[EMAIL PROTECTED]> wrote: On Mon, 25 Apr 2005, Randy Bush wrote: > i don't think this > > roam.psg.com:/usr/home/randy> doc -p -w worldnic.net > Doc-2.1.4: doc -p -w worldnic.net > Doc-2.1.4: Starting test of worldnic.net. parent is net. > Doc-2.1.4: Test date - Mon Apr 25 14:20:45 HST 2005 > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @a.gtld-servers.net. for NS of worldnic.net. failed > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @b.gtld-servers.net. for NS of worldnic.net. failed > > is the worldnic problem, but could be. but it is a problem. a.gtld-servers.net and b.gtld-servers.net have records. Some applications and stacks try the v6 address first if it's available and will appear to hang if you don't have v6 connectivity. That may very well be what's happening here. Matt -- Matt Larson <[EMAIL PROTECTED]> VeriSign Naming and Directory Services -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet [EMAIL PROTECTED] or [EMAIL PROTECTED] ferg's tech blog: http://fergdawg.blogspot.com/
Re: Problems with NS*.worldnic.com
On Mon, 25 Apr 2005, Randy Bush wrote: > i don't think this > > roam.psg.com:/usr/home/randy> doc -p -w worldnic.net > Doc-2.1.4: doc -p -w worldnic.net > Doc-2.1.4: Starting test of worldnic.net. parent is net. > Doc-2.1.4: Test date - Mon Apr 25 14:20:45 HST 2005 > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @a.gtld-servers.net. for NS of worldnic.net. failed > ;; res_nsend: Protocol not supported > DIGERR (UNKNOWN): dig @b.gtld-servers.net. for NS of worldnic.net. failed > > is the worldnic problem, but could be. but it is a problem. a.gtld-servers.net and b.gtld-servers.net have records. Some applications and stacks try the v6 address first if it's available and will appear to hang if you don't have v6 connectivity. That may very well be what's happening here. Matt -- Matt Larson <[EMAIL PROTECTED]> VeriSign Naming and Directory Services
Re: Problems with NS*.worldnic.com
something *very* strange is going on. the worldnic servers have been giving delayed or no results for days now. and nsi is hoping we and the wsj/nyt won't notice. i don't think this roam.psg.com:/usr/home/randy> doc -p -w worldnic.net Doc-2.1.4: doc -p -w worldnic.net Doc-2.1.4: Starting test of worldnic.net. parent is net. Doc-2.1.4: Test date - Mon Apr 25 14:20:45 HST 2005 ;; res_nsend: Protocol not supported DIGERR (UNKNOWN): dig @a.gtld-servers.net. for NS of worldnic.net. failed ;; res_nsend: Protocol not supported DIGERR (UNKNOWN): dig @b.gtld-servers.net. for NS of worldnic.net. failed is the worldnic problem, but could be. but it is a problem. (i generally ignore b root issues). but it's probably time for us all to dump symptoms here and figure it out as a community, as the dog with the bone ain't 'fessing up. randy
RE: Problems with NS*.worldnic.com
On Mon, 25 Apr 2005, Graeme Clark wrote: > > > > >I saw some mention of this in a previous thread. Is anyone else still > >experiencing problems? We're seeing general slowness and the use of the > >truncate bit in responses, forcing to TCP mode. > > > We're still having a wack of issues with all names on NSI nameservers. Poking > around at other service provider nameservers out there I'm amazed at how many > places things are still resolving, something to be said for ignoring DNS > TTLs. :) It's been over 3 days now, so I think I'm going to move what I've > still got left with them somewhere else. > > I'm getting a roughly 10% response rate from their nameservers, and that's > probably optimisticly high. it may help 'other operators' and 'nsi' to know which servers you can NOT resolve from, and which cache/recursive hosts are asking ns.worldnic.com for the particular domains. So: I am at 128.2.35.50, I asked cache00.ns.uu.net for a domain on both: ns5.worldnic.com and ns25.worldnic.com and got no response :( (as a simple for instance, this may help others know they are not alone in their problems, and the NSI folks might know which askers and answers are still unhappy with each other)
RE: Problems with NS*.worldnic.com
>I saw some mention of this in a previous thread. Is anyone else still >experiencing problems? We're seeing general slowness and the use of the >truncate bit in responses, forcing to TCP mode. We're still having a wack of issues with all names on NSI nameservers. Poking around at other service provider nameservers out there I'm amazed at how many places things are still resolving, something to be said for ignoring DNS TTLs. :) It's been over 3 days now, so I think I'm going to move what I've still got left with them somewhere else. I'm getting a roughly 10% response rate from their nameservers, and that's probably optimisticly high. Graeme graeme clark | leader, network technologies and services | lavalife corp. toronto | tel 416 263 6300 x 3658 | fax 416 263 6303
Re: Problems with NS*.worldnic.com
We have few servers with Interland / Miami. Today for around 1 hour 15 minutes the dns / tcp traffic was timing out. Httpd was very slow for domains with backup dns servers in Europe but other domains with DNS within Interland only was not resolving at all. I only noticed that traffic was not going through from here, Saudi Arabia but it appeared to be resolving okay from United States. I do not know if this is related to worldnic dns problem, but I think Interland is outsourcing DNS from Verisign. [EMAIL PROTECTED] - Original Message - From: "Greg Schwimer" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Monday, April 25, 2005 21:34 Subject: Problems with NS*.worldnic.com > > I saw some mention of this in a previous thread. Is anyone else still > experiencing problems? We're seeing general slowness and the use of the > truncate bit in responses, forcing to TCP mode. >
