Any time you deal with pushing files around, you also have to deal with the repercussions of when the file fails to get to its destination or it fails to get there in a timely manner. [Hai hadoop config files.] If you use an interface alias/vip/multi-a/whatever to deal with namenode availability, then the host information becomes even more critical.
Rather than build something custom, I chose to use well known, off the shelf software to deal with keeping host information relatively in-sync: bind. On 10/19/09 8:09 PM, "David B. Ritch" <[email protected]> wrote: > Most of the communication and name lookups within a cluster refer to > other nodes within that same cluster. It is usually not a big deal to > put all the systems from a cluster in a single hosts file, and rsync it > around the cluster. (Consider using prsync, which comes with pssh, > http://www.theether.org/pssh/, or your favorite cluster management > software.) > Editing each individually clearly doesn't scale; but editing it once and > replicating it does. > > Is a large hosts file less efficient than nscd or a caching DNS server > for nodes within the cluster? > > Thanks, > > David > > On 10/19/2009 8:02 PM, Edward Capriolo wrote: >> On Mon, Oct 19, 2009 at 7:17 PM, Allen Wittenauer >> <[email protected]> wrote: >> >>> >>> >>> On 10/19/09 11:46 AM, "Edward Capriolo" <[email protected]> wrote: >>> >>> >>>> I am interested in your post. What has caused you to run caching DNS >>>> servers on each of your nodes? Is this a hadoop specific problem or a >>>> problem specific to your implementation? >>>> >>> Hadoop does a -tremendous- amount of hostname lookups. If you don't have >>> either nscd or a local DNS caching server, you are likely throwing what >>> could be some significant performance gains away. >>> >>> >>>> My assumption here is that a hadoop cluster of say 1000 nodes would >>>> repeatedly talk to the same 1000 nodes. >>>> >>> ... and that's the catch! Every node running the DFSClient code or being >>> called out from a map/reduce task is a potential hostname that would need be >>> resolved. Just think about something like distcp. >>> >>> Also note that this is before we talk about monitoring, any other naming >>> services, CNAMEs, multi-As, etc, that get built as a normal part of running >>> an infrastructure. >>> >>> >>>> Are you saying that nscd is >>>> inadequacy to handle the size of the cache, or nscd is not very >>>> efficient? What exactly is the reason you are running a caching DNS >>>> server on each node? >>>> >>> In the case of Yahoo!, we had (or, at least, a perception) that we had or >>> were going to have jobs that did a lot of direct DNS lookups and/or >>> accessed/referenced things outside of the local grid. Also note that a DNS >>> caching server is going to store more information about hostnames than a >>> simple host to IP service like nscd. >>> >>> Hypothetical: Let's say I'm building rules for a spam filter and part of my >>> process is to look up the MX record for a given host. nscd isn't going to >>> help you there. >>> >>> In the case of LinkedIn, the jury is still out. I suspect we don't have >>> nscd.conf tuned correctly. Our grid isn't that big, our connections in/out >>> are fairly small, etc. It has been one of the things on my todo list since I >>> got hired here 2 months ago. :) >>> >>> [For the record, I'm not one of those crazy people who turns off nscd >>> because I had a bad experience with a broken version five years ago. In >>> the case of Yahoo!, I was the crazy person who started insisting we turn it >>> on, albeit not for hosts.] >>> >>> >>> >> Cool thanks for the info. >> >> I have found NSCD to be absolutely essential in most/all situations. >> Whenever I would truss processes on OS'es without NSCD (say freebsd >> 6.2) I would see numerous repeated 'stat' against /etc/passwd and >> /etc/group. >> >> If you are doing users and groups through LDAP nscd is super important >> as well. Your not going to want to make a series of lookups each stat. >> >> I would think the most efficient implementation would be nscd and a >> local caching server in that case. NSCD should be very efficient since >> it is done through libraries, dns lookups have to open sockets >> (overhead). However I can see your point nscd can not do other types >> of records. >> >> >
