David B. Ritch wrote:
Most of the communication and name lookups within a cluster refer to
other nodes within that same cluster.  It is usually not a big deal to
put all the systems from a cluster in a single hosts file, and rsync it
around the cluster.  (Consider using prsync, which comes with pssh,
http://www.theether.org/pssh/, or your favorite cluster management
software.)
Editing each individually clearly doesn't scale; but editing it once and
replicating it does.

Is a large hosts file less efficient than nscd or a caching DNS server
for nodes within the cluster?


Pro
 * removes the DNS server as a SPOF
 * works on clusters without DNS servers (virtual ones, for example)
* lets you set up private hostnames ("namenode", "jobtracker") that don't change
 * lets you keep the cluster config under SCM

Con
 * harder to push out changes
 * wierd errors when your cluster is inconsistent


We could do a lot in Hadoop in detecting and reporting DNS problems; contributions here would be very welcome. They are a dog to test though.

Reply via email to