Re: large organization nameservers sending icmp packets to dns servers.
On Thu, 09 Aug 2007 22:58:40 -, Paul Vixie said: How does the (eventual) deployment of DNSSEC change these numbers? DNSSEC cannot be signalled except in EDNS. Right. Elsewhere in this thread, somebody discussed ugly patches to keep the packet size under 512. I dread to think how many different ways of protecting DNS are deployed that will break EDNS, and just haven't been noticed because there's little enough *actual* EDNS breakage that it's down in the noise of *other* random voodoo breakage at those sites. And who's likely to feel *that* pain first? the DNSSEC design seems to distribute pain very fairly. I actually meant which 800 pound gorilla is going to try this first and find all the bustifications, but your answer is good too.. :) pgpFy1Ds4pwV8.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
On Thu, 9 Aug 2007 15:53:12 -0700 (PDT) Doug Barton [EMAIL PROTECTED] wrote: How many bytes of shell code can you stuff into a 4096 byte EDNS0 UDP packet? :) Probably a lot. People used to have 4-line signatures with the PGP encryption or DECSS. I have a 152-byte C program that calculates 32K digits of PI. matthew black network services california state university, long beach
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 9, 2007, at 2:05 PM, Paul Vixie wrote: Your comments have helped. i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result. Monitoring is a good recommendation, as many incorrectly estimate record limits. Wildcard resources should also be checked against maximal labels. Fallback may occur with resource records encompassing a bit more than a couple hundred bytes. The assurance TCP will fail first is heartening. How this protection interacts with an emerging exploit could be interesting. I'll try to setup some tests and be less pragmatic. or perhaps you're asking that EDNS be more widely implemented, that it not be blocked by firewalls or perverted by hotelroom DNS middleboxes, and that firewalls start allowing UDP fragments (which don't have port numbers and therefore won't be allowed by UDP/53 rules). TCP offers a means to escape UDP related issues. On the other hand, blocking TCP may offer the necessary motivation for having these UDP issues fixed. After all, only UDP should be required. When TCP is designed to readily fail, reliance upon TCP seems questionable. As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled. UDP handling may have been easier had EDNS been limited to 1280 bytes. On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38. No pain, no gain might be a motto that applies equally to DNS as it does for weight lifting. -Doug
Re: large organization nameservers sending icmp packets to dns servers.
Your comments have helped. groovy. When TCP is designed to readily fail, reliance upon TCP seems questionable. i caution against being overly cautious about DNS TCP if you're using RFC 1035 section 4.2.2 as your basis for special caution. DNS TCP only competes directly against other DNS TCP. there are only two situations where a DNS TCP state blob is present in a DNS target (server) long enough to be in any danger: when doing work upstream to fulfill the query, and in zone transfers. when answering DNS TCP queries in an authority server, there is by definition no upstream work to be done, other than possible backend database lookups which are beyond the scope of this discussion. these responses will usually be generated synchronous to the receipt of the last octet of a query, and the response will be put into the TCP window (if it's open, which it usually is), and the DNS target (server) will then wait for the initiator (client) to close the connection or send another query. (usually it's a close.) when answering DNS TCP zone transfer requests in an authority server, there is a much larger window of doom, during which spontaneous network congestion can close the outgoing TCP window and cause a DNS target (server) to think that a TCP session is idle for the purpose of RFC 1035 section 4.2.2 TCP resource management. while incremental zone transfer is slightly less prone to this kind of doom than full zone transfer, since the sessions are shorter, it can take some time for the authority server to compute incremental zone diffs, during which the TCP session may appear idle through no fault of the DNS initiator (client) who is avidly waiting for its response. lastly, when answering DNS TCP queries in a recursive caching nameserver, it can take a while (one or more round trips to one or more authority servers) before there is enough local state to satisfy the query, during which time the TCP resources held by that query might be reclaimed under RFC 1035 section 4.2.2's rules. the reason why not to be overly cautious about TCP is that in the case where you're an authority server answering a normal query, the time window during which network congestion could close the outbound TCP window long enough for RFC 1034 section 4.2.2's rules to come into effect, is vanishingly short. so while it's incredibly unwise to depend on zone transfer working from a small number of targets to a large number of initiators, and it is in fact wise to firewall or ACL your stealth master server so that only your designated secondary servers can reach it, none of this comes into play for normal queries to authority servers -- only zone transfers to authority servers. the unmanageable risk is when a recursive caching nameserver receives a query by TCP and forwards/iterates upstream. if this happens too often, then the RFC 1035 section 4.2.2 rules will really hurt. and thus, it's wise, just as you say, to try to make sure other people don't have to use TCP to fetch data about your zone. the counterintuitive thing is that you won't be able to measure the problems at your authority server since that's not where they will manifest. they'll manifest at caching recursive servers downstream. As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled. this would be true if TCP fallback was used when EDNS failed. it's not. if EDNS fails, then EDNS will not be used, either via UDP or TCP. so if improper handling of UDP prevents EDNS from working, then EDNS and anything that depends on EDNS, including DNSSEC, will not be used. UDP handling may have been easier had EDNS been limited to 1280 bytes. if you mean, had EDNS been limited to nonfragmentation cases, then i think you might mean 576 bytes or even 296 bytes. 1280 is an IPv6 (new era) limit. On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38. i surely do hope so. we need those ACLs and we need that deployment, and if message size and TCP fallback is a motivator, then let's turn UP the volume.
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 10, 2007, at 4:41 PM, Paul Vixie wrote: On the other hand, potentially larger messages may offer the necessary motivation for adding ACLs on recursive DNS, and deploying BCP 38. i surely do hope so. we need those ACLs and we need that deployment, and if message size and TCP fallback is a motivator, then let's turn UP the volume. There are so many more larger and immediate reasons for doing these things that I seriously doubt message size and TCP fallback on the DNS will have any impact at all in terms of motivating the non- motivated. But, one can always hope. ; --- Roland Dobbins [EMAIL PROTECTED] // 408.527.6376 voice Culture eats strategy for breakfast. -- Ford Motor Company
Re: large organization nameservers sending icmp packets to dns servers.
On 8/9/2007 at 10:07 PM, Mark Andrews [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED] you write: I suspect that the origin of the myth that DNS/TCP is more dangerous than DNS/UDP is that the first root expliot of named was over TCP not UDP. There were later exploits that were UDP only which totally busted the myth but it continues to live. Mark Just to make it clear. This was BIND 4/8 code and the bugs were addressed in the last millennia. To date there are no known root exploits for BIND 9. Because who runs BIND as root anymore? Lots of people. It's the only way you can handle some events. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: [EMAIL PROTECTED]
Re: large organization nameservers sending icmp packets to dns servers.
On Fri, 10 Aug 2007 16:11:04 -0700 Douglas Otis [EMAIL PROTECTED] wrote: TCP offers a means to escape UDP related issues. On the other hand, blocking TCP may offer the necessary motivation for having these UDP issues fixed. After all, only UDP should be required. When TCP is designed to readily fail, reliance upon TCP seems questionable. As DNSSEC in introduced, TCP could be relied upon in the growing number of instances where UDP is improperly handled. As a datapoint I ran some tests against a reasonably diverse and sizeable TLD zone I work with in another forum. I queried the name servers listed in the parent to see if I could successfuly query them for their corresponding domain name they are configured for using TCP. Out of about 9,300 unique name servers I failed to receive any answer from about 1700 of them. That is a bit more than an 18% failure rate. John
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 8, 2007, at 5:35 PM, Paul Vixie wrote: ... but a TCP connection will consume a significant amount of a name server's resources. ...wrong. Wanting to understand this comment, ... the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets. The relevant entry in Section 1035 4.2.2 recommends that the server not block other activities waiting for TCP data. This is not exactly a requirement that TCP should fail before UDP. The concern leading to a suggestion that TCP always fail was a bit different. A growing practice treats DNS as a type of web server when used to publish rather bulky script-like resource records. Due to typical sizes, it is rather common to find these records depend upon TCP fallback. This problem occurred with paypal, for example. TCP fallback is especially problematic when these records are given wildcards. Such fallback increases the amplification associated with an exploit related to the use of the script within the record. Of course there are better ways to solve this problem, but few are as certain. -Doug
Re: large organization nameservers sending icmp packets to dns servers.
the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets. The relevant entry in Section 1035 4.2.2 recommends that the server not block other activities waiting for TCP data. This is not exactly a requirement that TCP should fail before UDP. it is semantically equivilent to such a requirement, in that UDP/53 is an other activity performed by name servers. it happens to be implemented this way in all versions of BIND starting in 4.8 or so (when named-xfer was made a separate executable), all versions of Windows DNS, and all current name server implementations i am aware of (including powerdns, nominum ANS, and NSD). so while not exactly, it's effectively a requirement, and i think we ought to design our systems with this constraint as a given. The concern leading to a suggestion that TCP always fail was a bit different. A growing practice treats DNS as a type of web server when used to publish rather bulky script-like resource records. Due to typical sizes, it is rather common to find these records depend upon TCP fallback. This problem occurred with paypal, for example. TCP fallback is especially problematic when these records are given wildcards. Such fallback increases the amplification associated with an exploit related to the use of the script within the record. Of course there are better ways to solve this problem, but few are as certain. i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result. or perhaps you're asking that EDNS be more widely implemented, that it not be blocked by firewalls or perverted by hotelroom DNS middleboxes, and that firewalls start allowing UDP fragments (which don't have port numbers and therefore won't be allowed by UDP/53 rules). i would agree with either recommendation. but i won't agree that TCP creates stability or load problems for servers.
Re: large organization nameservers sending icmp packets to dns servers.
On Thu, 09 Aug 2007 21:05:26 -, Paul Vixie said: i think you're advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result. How does the (eventual) deployment of DNSSEC change these numbers? And who's likely to feel *that* pain first? pgpjVn8NTTT7U.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
[EMAIL PROTECTED] writes: ... advising folks to monitor their authority servers to find out how many truncated responses are going out and how many TCP sessions result from these truncations and how many of these TCP sessions are killed by the RFC1035 4.2.2 connection management logic, and if the numbers seem high, then they ought to change their applications and DNS content so that truncations no longer result. How does the (eventual) deployment of DNSSEC change these numbers? DNSSEC cannot be signalled except in EDNS. And who's likely to feel *that* pain first? the DNSSEC design seems to distribute pain very fairly. -- Paul Vixie
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 6 Aug 2007, Drew Weaver wrote: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization? If someone else mentioned this I didn't see it, so I think it's worthwhile to point out that in the case of Yahoo! at least the pinging isn't done by nsX.yahoo.com, but rather it's done by Akamai after you get to the point in the chain where the real answer is going to come from one of their servers. Someone else already did point out that in spite of the existence of outliers, the shape of the bell-shaped curve remains the same, and if this method didn't actually work then it wouldn't be in such widespread use. hth, Doug -- If you're never wrong, you're not trying hard enough
Re: large organization nameservers sending icmp packets to dns servers.
I suspect that the origin of the myth that DNS/TCP is more dangerous than DNS/UDP is that the first root expliot of named was over TCP not UDP. There were later exploits that were UDP only which totally busted the myth but it continues to live. Mark
Re: large organization nameservers sending icmp packets to dns servers.
In article [EMAIL PROTECTED] you write: I suspect that the origin of the myth that DNS/TCP is more dangerous than DNS/UDP is that the first root expliot of named was over TCP not UDP. There were later exploits that were UDP only which totally busted the myth but it continues to live. Mark Just to make it clear. This was BIND 4/8 code and the bugs were addressed in the last millennia. To date there are no known root exploits for BIND 9. Mark
Industry best practices (was Re: large organization nameservers sending icmp packets to dns servers)
On Tue, 7 Aug 2007, Kevin Oberman wrote: This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the standard practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs. Having worked on both sides of the fence, i.e. I was a card-carrying member of both ASIS and NFPA, I used grumbled about the kooky things sysadmins and programmers did in the name of security as much as I grumbled about the kooky things security folks did in the name of security. Heck, if programmers only produced bug-free software and sysadmins kept only well configured systems, security people would have a lot less work to do. What are the industry best practices for keeping DNS servers secure? CERT publishes a document on securing DNS: http://www.cert.org/archive/pdf/dns.pdf NIST publishes a document on securing DNS: http://csrc.nist.gov/fasp/FASPDocs/network-security/NISTSecuringDNS.htm CMYRU publishes a document on securing DNS: http://www.cymru.com/Documents/secure-bind-template.html Microsoft publishes a document on securing DNS: http://technet2.microsoft.com/WindowsServer/en/Library/0fe406eb-6ca2-4d95-9a18-aede7e931ca21033.mspx IETF publishes a document on operational (including security) requirements for root DNS servers: http://www.rfc-editor.org/rfc/rfc2870.txt While there is a lot in common, they each also have variations and omissions. Especially when it comes to some possibly obscure interactions with many different protocols and applications. The relationships between IP, ICMP, TCP, UDP and DNS seems to be tough for many people to get right. When you add undocumented common knowledge and other applications leveraging DNS for all sorts of stuff besides name/address resolution, its the typical programmer generated pile of spaghetti. Its often simplier to wait for something to break before you fix it. I know many sysadmins, programmers and even security people, that use that philosphy to decide which things to work on today. The good thing about security folks (and their cousins, the auditors) is most are compliance driven. So if you get a new Industry Best Practice, often they will become your friend enforcing whatever that says. So what should the Industry Best Practice(s) for DNS servers (root, authoritative and recursive) be? And what should it say about the interaction between IP/ICMP and TCP/UDP? And maybe we'll even get G-Root to follow it.
Re: large organization nameservers sending icmp packets to dns servers.
i normally agree with doug [EMAIL PROTECTED] (Douglas Otis) writes: Ensuring an authoritative domain name server responds via UDP is a critical security requirement. TCP will not create the same risk of a resolver being poisoned, but a TCP connection will consume a significant amount of a name server's resources. ...but this is flat out wrong, dead wrong, no way to candy coat it, wrong. -- Paul Vixie
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 8, 2007, at 12:11 PM, Paul Vixie wrote: [EMAIL PROTECTED] (Douglas Otis) writes: Ensuring an authoritative domain name server responds via UDP is a critical security requirement. TCP will not create the same risk of a resolver being poisoned, but a TCP connection will consume a significant amount of a name server's resources. ...but this is flat out wrong, dead wrong, no way to candy coat it, wrong. Wanting to understand this comment, I'll expand upon the quoted statement. Resolver's factors affecting DNS security are: - selection of port and transaction IDs - restrictions on outstanding queries for same resource - limits on inbound bandwidth Ignoring uncontrollable factors... Authoritative server factors affecting security are: - time frame for an answer - duration of RR TTLs - number of servers A short time frame for an answer along with longer TTLs are influenced by authoritative servers and also affect spoofing rates. When DNS TCP is used, the transport sequence number further precludes a spoofed TCP answer from being accepted. When a truncated response is returned, TCP fallback may be attempted. When a TCP ICMP refusal is filtered or never sent, but TCP has been blocked, the timeframe alloted for spoofing could entail the entire TCP timeout. However, probability for successful spoofing includes an additional multiplier of 1 / 2^32. This reduction should sufficiently negate an additional timeout duration. TCP requires state and introduces several additional exchanges for a given number of answers. Any effort related to poisoning will likely attempt to delay an answer by adding to the server's overhead. Precluding truncation, and thereby eliminating TCP, should favorably reduce server overhead and increase overall performance. Of course, a more practical method would be to ensure sufficient DNS resources are available by increasing server resources. That said, how many domains allocate a couple of prior generation servers for DNS? -Doug
Re: large organization nameservers sending icmp packets to dns servers.
... but a TCP connection will consume a significant amount of a name server's resources. ...wrong. Wanting to understand this comment, ... the resources given a nameserver to TCP connections are tightly controlled, as described in RFC 1035 4.2.2. so while TCP/53 can become unreliable during high load, the problems will be felt by initiators not targets. (this is why important AXFR targets have to be firewalled down to a very small population of just one's own nameservers, and is why important zones have to use unpublished primary master servers, and is why f-root's open AXFR of the root zone is a diagnostic service not a production service.)
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 8, 2007, at 6:20 PM, william(at)elan.net [EMAIL PROTECTED] wrote: On Tue, 7 Aug 2007, Donald Stahl wrote: All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid. Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted. How is that an anti DoS technique when you actually need to return an answer via UDP in order to force next request via TCP? Or is this techinque based on premise that an attacker will not spoof packets and thus will send flood of DNS requests to server from same IP (set of ips)? If so the result would be that attacker could in fact use TCP just as well as UDP. The anti-ddos box sends back a UDP reply with the TCP bit sent and no data. Which, I believe, violates the RFC. (But it is too hard to look up on my iPhone. :) If so, guess that makes those boxes 'stupid'. -- TTFN, patrick
RE: large organization nameservers sending icmp packets to dns servers.
All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid. Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted. In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems. -Don
Re: large organization nameservers sending icmp packets to dns servers.
Hi Donald, I'm not prepared to call it stupid, but you're right it can cause issues. -J Sent via BlackBerry - Original Message - From: Donald Stahl [EMAIL PROTECTED] To: Jason J. W. Williams Cc: [EMAIL PROTECTED] [EMAIL PROTECTED]; John Levine [EMAIL PROTECTED]; [EMAIL PROTECTED] [EMAIL PROTECTED] Sent: Tue Aug 07 12:14:11 2007 Subject: RE: large organization nameservers sending icmp packets to dns servers. All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid. Several anti DoS utilities force unknown hosts to initiate a query via TCP in order to be whitelisted. If the host can't perform a TCP query then they get blacklisted. In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems. -Don !SIG:46b8b686156533728213125!
Re: large organization nameservers sending icmp packets to dns servers.
From: Joe Abley [EMAIL PROTECTED] Date: Tue, 7 Aug 2007 15:19:30 -0400 Sender: [EMAIL PROTECTED] On 7-Aug-2007, at 14:38, Patrick W. Gilmore wrote: On Aug 7, 2007, at 2:14 PM, Donald Stahl wrote: All things being equal (which they're usually not) you could use the ACK response time of the TCP handshake if they've got TCP DNS resolution available. Though again most don't for security reasons... Then most are incredibly stupid. Those are impressively harsh words. But they are hard to argue with. In addition, any UDP truncated response needs to be retried via TCP- blocking it would cause a variety of problems. Since we are talking about authorities here, one can control the size of ones responses. Never reply with anything big and hence never set TC seems like a reasonable, expedient way to circumvent the problem of wholesale 53/ tcp-blocking stupidity. It doesn't make the behaviour any less stupid, though. The security argument looks even more bizarre when you consider what the DO bit in a request will do in general to the size of a response, in the case of an authority server which has signed zone data. This has been a pain for me for years. I have tried to reason with security people about this and, while they don't dispute my reasoning, they always end up saying that it is the standard practice and that, lacking any evidence of what it might be breaking, it will continue to be blocked. And I don't mean small companies, either. One of the biggest issues I have is with one of the countries largest government funded research labs. Wonder how often DNSSEC might make non-transfer queries tickle this and really break things? (Assuming we ever get wide use of DNSSEC.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 pgpU18YPgK72t.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 7, 2007, at 4:33 PM, Donald Stahl wrote: [...] If you don't like the rules- then change the damned protocol. Stop just doing whatever you want and then complaining when other people disagree with you. I think this last part is the key. Remember the old adage: My network, My rules? Have we forgotten that? Should I not block ports for MS protocols when a new worm spreads because it would break the E-2-E principal? What about spam filtering? Or a myriad of other things. Everyone here is breaking some RFC somehow. And most of us don't give a rats ass. Which is the way it should be. But when you decide that YOUR violation is MY problem to fix, then you are just being silly. And worse, annoying. Let's all just agree to run our own networks the way we damned well please, as long as we are not hurting anyone else. We just have to define omplaining to you about things I b0rk'ed by myself as hurting you. Which isn't a stretch, support costs money, and costing me money because you screwed up is definitely hurtful. -- TTFN, patrick P.S. To be clear, no, your pr0n site not resolving because I don't support TCP does not qualify as hurting you unless I call you to complain about it, even if it loses you revenue.
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest. Yes, it's a semi-borkken strategy, because it assumes that: 1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path slow when it really isn't, just becase a router slow-pathed ICMP). 2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup. pgpq1zcxucApB.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
On Monday 06 August 2007 16:53, Drew Weaver wrote: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? Some of the DNS load balancing schemes do this, I assume to work out how far away your server is so they can give geographically relevant answers. If you are geographically close to your recursive name server, it might even work. And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization? I tend to favour filtering some types of ICMP packets and not others, the packets required for ping hold little fear for me (and are kind of useful), but YMMV. My ICMP filtering experience is not DNS specific, you might be able to do better with DNS server specific rule, but that is too much like micromanagement for me, others may know a lot more on this.
Re: large organization nameservers sending icmp packets to dns servers.
Drew Weaver wrote: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization? Wearing my Mozilla hat here... I blogged about this (blog.mozilla.com/mrz, somewhere there) and Asa blog'd about it over at http://weblogs.mozillazine.org/asa/archives/2007/03/trying_to_load.html . Mozilla uses Citrix Netscalers and we're currently using dynamic proximity for load balancing between data centers. After Asa's post, we found poorly documentation that led to misconfiguration of the probe settings. I've cut down the number of probes (default was icmp, udp and tcp:80 to a nameserver) and instead of the ~10 complaints a day I was getting, I get many one a month. If you're still annoyed by the probes, ping me off-list. - mz
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 06 Aug 2007 11:57:08 -0400 [EMAIL PROTECTED] wrote: On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest. Yes, it's a semi-borkken strategy, because it assumes that: 1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path slow when it really isn't, just becase a router slow-pathed ICMP). This is aimed at hosts, not routers, right? As far as I know, routers don't slow-path forwarded ICMP. Hosts will probably reply to ICMP from their kernel, so it's a faster response than a user-level DNS reply. 2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup. Right. But there's no particular reason to block it, unless the rate is high enough that it's causing you CPU or network load problems. (If it's your IDS that's getting overloaded, perhaps tell it not to worry unless you see other load issues...) --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: large organization nameservers sending icmp packets to dns servers.
But why would they care where the nameserver is? Point 2 would seem to be a little stupid a thing to assume. Also, what happens if, at that moment, the ICMP packet is stuck in a queue for a few ms making the shortest route longer. -- Leigh [EMAIL PROTECTED] wrote: On Mon, 06 Aug 2007 11:53:15 EDT, Drew Weaver said: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? Sounds like one of the global-scale load balancers - when you do a (presumably) recursive DNS lookup of one of their hosts, they'll ping the nameserver from several locations and see which one gets an answer the fastest. Yes, it's a semi-borkken strategy, because it assumes that: 1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path slow when it really isn't, just becase a router slow-pathed ICMP). 2) That the actual requester of service is reasonably near net-wise to the server handling the end-user's recursive DNS lookup.
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 6, 2007, at 9:13 AM, Leigh Porter wrote: But why would they care where the nameserver is? Point 2 would seem to be a little stupid a thing to assume. Also, what happens if, at that moment, the ICMP packet is stuck in a queue for a few ms making the shortest route longer. While point 2 is a bad assertion if you depend completely upon it, it's not necessarily a bad starting point if you have no other data to go on. 1. 90+% of resolvers are topologically proximate to either the requestor, or, the requestors NAT box that you will have to talk to anyway. 2. At the GLB level, you really don't have any data other than the IP address of the resolver upon which to base your GLB decision. Since you'll be right 90+% of the time, and, only sub-optimal, not broken the other 10% of the time, it generally works OK. 3. When I worked for Netli, before they were acquired in what I would call a much less than ethical transaction, we maintained an exception table for cases where we learned that the DNS resolver was not topologically proximate to the requestors that flowed through it. We also spent a fair amount of time explaining the benefits of having the resolver be topologically proximate to our customers and their customers. The Netli system was designed to be quite gentle in the amount of probing it did, but, we did occasionally get messages from people with paranoid IDS boxes. Usually, once we explained that our efforts were directed at improving the quality of service to their users, and how the system worked and how little traffic we sent their way to accomplish this, they were happy to reconfigure their alarm preferences. I don't have first hand knowledge of anyone elses use of these kinds of ICMP probes, but, I would say that generally, they are somewhat useful and mostly harmless. Owen
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 06 Aug 2007 12:13:03 EDT, Steven M. Bellovin said: 1) ICMP is handled at the same rate as TCP/UDP packets in all the routers involved (so there's no danger of declaring a path slow when it really isn't, just becase a router slow-pathed ICMP). This is aimed at hosts, not routers, right? As far as I know, routers don't slow-path forwarded ICMP. Hosts will probably reply to ICMP from their kernel, so it's a faster response than a user-level DNS reply. Well, they don't *directly* slow-path it. But we've seen *plenty* of cases of multi-hop performance as indicated by ICMP Echo Request/Reply doesn't at all match throughput/latency as indicated by TCP-level stats mentioned on this list... pgpA3RXyxDK49.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
Why would they ping rather than just sending the query to all of the NS and see which one answers first? It's an IP round trip either way. If you have sites in San Fran, London, and Tokyo, and you launch a ping from all 3 and see which one gets there first, you'll *know* the RTT from each site. If you just send DNS replies from all 3, you don't have a good way of telling which one got to the destination first. Um, unless I seriously misunderstand the client DNS cache wants to know which server is closest. So it sends DNS queries to all three NS at the same time. Then it waits for the answers. Whichever one answers first is the closest. What am I missing? Regards, John Levine, [EMAIL PROTECTED], Primary Perpetrator of The Internet for Dummies, Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor More Wiener schnitzel, please, said Tom, revealingly.
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 06 Aug 2007 16:11:36 EDT, Matthew Crocker said: But you could, it isn't hard to dump a BGP view into a box from a border router and use that map to determine the proper DNS records to return. It's harder than it looks, given the number of people who pop up on this list and ask how to get BGP to Do The Right Thing when 2 paths are the same length but vastly differing in bandwidth, number-of-actual-hops, and/or latency pgpi11qlEUAjq.pgp Description: PGP signature
Re: large organization nameservers sending icmp packets to dns servers.
On Mon, 6 Aug 2007, Drew Weaver wrote: Is it a fairly normal practice for large companies such as Yahoo! And Mozilla to send icmp/ping packets to DNS servers? If so, why? And a related question would be from a service provider standpoint is there any reason to deny ICMP/PING packets to name servers within your organization? They use ICMP/Echo Request to calculate a rough surrogate latency estimate for potential users of that caching DNS server so they can return different DNS answers depending on your network topology. Yes its an approximation, and can be wrong. Some networks even re-route ICMP/Echo to a completely different box which just responsed to pings; so it may not even go to the same place. Given all those caveats, many times its still the best guess you can make. ICMP/ECHO is a separate protocol which is easy to filter if you want to, without affecting normal TCP/UDP/etc packets. But then expect to get worse default DNS answers from those same sites attempting to optimize their DNS answers. It would be cool if people ran NTP port 123 on their DNS servers, and then we could get extreme measurements. But then I'm sure someone would point out 62 flaws with that. In the end, its up to each network operator to make its own decision. If your DNS servers aren't being negatively impacted, and it helps your users get better answers, you might keep it. If the answers are reversed, you might drop them. My IDS is badly tuned Well maybe there is a fix for that.
Re: large organization nameservers sending icmp packets to dns servers.
On Aug 6, 2007, at 4:43 PM, [EMAIL PROTECTED] wrote: On Mon, 06 Aug 2007 16:11:36 EDT, Matthew Crocker said: But you could, it isn't hard to dump a BGP view into a box from a border router and use that map to determine the proper DNS records to return. It's harder than it looks, given the number of people who pop up on this list and ask how to get BGP to Do The Right Thing when 2 paths are the same length but vastly differing in bandwidth, number-of-actual-hops, and/or latency Plus you are assuming BGP knows the right answer. People here are arguing ICMP Echo is a bad metric. BGP makes ICMP look like the gold standard. -- TTFN, patrick
