Everything can get made to work in a small scale. As the grid grows, well...
On 10/20/09 10:32 AM, "David Ritch" <[email protected]> wrote: > I also prefer to avoid custom software, and follow standards. We use Puppet > to manage our node configuration (including hadoop config files), and adding > one more file to the configuration is trivial. > > I prefer not to run additional daemons on all my nodes when I can avoid it. > Replicating our hosts file allows us to avoid running named on all the > nodes. > > David > > On Tue, Oct 20, 2009 at 1:15 PM, Allen Wittenauer > <[email protected]>wrote: > >> Any time you deal with pushing files around, you also have to deal with the >> repercussions of when the file fails to get to its destination or it fails >> to get there in a timely manner. [Hai hadoop config files.] If you use an >> interface alias/vip/multi-a/whatever to deal with namenode availability, >> then the host information becomes even more critical. >> >> Rather than build something custom, I chose to use well known, off the >> shelf >> software to deal with keeping host information relatively in-sync: bind. >> >> >> On 10/19/09 8:09 PM, "David B. Ritch" <[email protected]> wrote: >> >>> Most of the communication and name lookups within a cluster refer to >>> other nodes within that same cluster. It is usually not a big deal to >>> put all the systems from a cluster in a single hosts file, and rsync it >>> around the cluster. (Consider using prsync, which comes with pssh, >>> http://www.theether.org/pssh/, or your favorite cluster management >>> software.) >>> Editing each individually clearly doesn't scale; but editing it once and >>> replicating it does. >>> >>> Is a large hosts file less efficient than nscd or a caching DNS server >>> for nodes within the cluster? >>> >>> Thanks, >>> >>> David >>> >>> On 10/19/2009 8:02 PM, Edward Capriolo wrote: >>>> On Mon, Oct 19, 2009 at 7:17 PM, Allen Wittenauer >>>> <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On 10/19/09 11:46 AM, "Edward Capriolo" <[email protected]> wrote: >>>>> >>>>> >>>>>> I am interested in your post. What has caused you to run caching DNS >>>>>> servers on each of your nodes? Is this a hadoop specific problem or a >>>>>> problem specific to your implementation? >>>>>> >>>>> Hadoop does a -tremendous- amount of hostname lookups. If you don't >> have >>>>> either nscd or a local DNS caching server, you are likely throwing what >>>>> could be some significant performance gains away. >>>>> >>>>> >>>>>> My assumption here is that a hadoop cluster of say 1000 nodes would >>>>>> repeatedly talk to the same 1000 nodes. >>>>>> >>>>> ... and that's the catch! Every node running the DFSClient code or >> being >>>>> called out from a map/reduce task is a potential hostname that would >> need be >>>>> resolved. Just think about something like distcp. >>>>> >>>>> Also note that this is before we talk about monitoring, any other >> naming >>>>> services, CNAMEs, multi-As, etc, that get built as a normal part of >> running >>>>> an infrastructure. >>>>> >>>>> >>>>>> Are you saying that nscd is >>>>>> inadequacy to handle the size of the cache, or nscd is not very >>>>>> efficient? What exactly is the reason you are running a caching DNS >>>>>> server on each node? >>>>>> >>>>> In the case of Yahoo!, we had (or, at least, a perception) that we had >> or >>>>> were going to have jobs that did a lot of direct DNS lookups and/or >>>>> accessed/referenced things outside of the local grid. Also note that a >> DNS >>>>> caching server is going to store more information about hostnames than >> a >>>>> simple host to IP service like nscd. >>>>> >>>>> Hypothetical: Let's say I'm building rules for a spam filter and part >> of my >>>>> process is to look up the MX record for a given host. nscd isn't going >> to >>>>> help you there. >>>>> >>>>> In the case of LinkedIn, the jury is still out. I suspect we don't >> have >>>>> nscd.conf tuned correctly. Our grid isn't that big, our connections >> in/out >>>>> are fairly small, etc. It has been one of the things on my todo list >> since I >>>>> got hired here 2 months ago. :) >>>>> >>>>> [For the record, I'm not one of those crazy people who turns off nscd >>>>> because I had a bad experience with a broken version five years ago. >> In >>>>> the case of Yahoo!, I was the crazy person who started insisting we >> turn it >>>>> on, albeit not for hosts.] >>>>> >>>>> >>>>> >>>> Cool thanks for the info. >>>> >>>> I have found NSCD to be absolutely essential in most/all situations. >>>> Whenever I would truss processes on OS'es without NSCD (say freebsd >>>> 6.2) I would see numerous repeated 'stat' against /etc/passwd and >>>> /etc/group. >>>> >>>> If you are doing users and groups through LDAP nscd is super important >>>> as well. Your not going to want to make a series of lookups each stat. >>>> >>>> I would think the most efficient implementation would be nscd and a >>>> local caching server in that case. NSCD should be very efficient since >>>> it is done through libraries, dns lookups have to open sockets >>>> (overhead). However I can see your point nscd can not do other types >>>> of records. >>>> >>>> >>> >> >>
