Hi,
I'm interested in running on-demand Hadoop jobs on a shared physical cluster
without relying on having hadoop permanently installed on the nodes. HOD seems
perfect for this, but I have a few questions that I can't seem to find the
answers to from the available documentation.
First, I am wondering how the allocation actually works. For example, it seems
like the hod interface allows me to specify which nodes to run on (in
cluster_dir/hadoop-site.xml", and how many nodes to run on (-o option in hod).
When I execute this command, do I have to know in advance that all the nodes in
my cluster_dir/hadoop-site.xml are available for HOD to use (i.e., HOD might
feel free to wipe anything on it)? And if the number of requested nodes have is
smaller than the hosts specified in hadoop-site.xml, how does HOD decide which
nodes to run on?
Also, HOD is advertised as: "a very useful tool for Hadoop developers and
testers who need to share a physical cluster for testing their own Hadoop
versions." Do these users need to actively coordinate which nodes they use for
their virtual cluster or will HOD coordinate the different virtual clusters
(i.e., each HOD virtual cluster is essentially a batch job on the physical
cluster, and uses Torque/Maui to schedule)?
Finally, if anybody has successfully (or maybe unsuccessfully) tried to use it,
I'd be very interested to hear how it has gone so far.
thanks,
Alvin
- Hadoop-On-Demand (HOD) newbie: question and curious about us... Alvin AuYoung
-