Craig,
Hello,
We have two HOD questions:
(1) For our current Torque PBS setup, the number of nodes requested by
HOD (-l nodes=X) corresponds to the number of CPUs allocated, however
these nodes can be spread across various partially or empty nodes.
Unfortunately, HOD does not appear to honour the number of processors
actually allocated by Torque PBS to that job.
Just FYI, at Yahoo! we've set torque to allocate separate nodes for the
number specified to HOD. In other words, the number corresponds to the
number of nodes, not processors. This has proved simpler to manage. I
forget right now, but I think you can make Torque behave like this (to
not treat processors as individual nodes).
For example, a current running HOD session can be viewed in qstat as:
104544.trmaster user parallel HOD 4178 8 -- -- 288:0 R
01:48
node29/2+node29/1+node29/0+node17/2+node17/1+node18/2+node18/1
+node19/1
However, on inspection of the Jobtracker UI, it tells us that node19
has "Max Map Tasks" and "Max Reduce Tasks" both set to 2, when I think
that for node19, it should only be allowed one map task.
While HOD does not do this automatically, please note that since you are
bringing up a Map/Reduce cluster on the allocated nodes, you can submit
map/reduce parameters with which to bring up the cluster when allocating
jobs. The relevant options are --gridservice-mapred.server-params (or -M
in shorthand). Please refer to
http://hadoop.apache.org/core/docs/r0.19.0/hod_user_guide.html#Options+for+Configuring+Hadoop
for details.
I believe that for each node, HOD should determine (using the
information in the $PBS_NODEFILE), how many CPUs for each node are
allocated to the HOD job, and then set
mapred.tasktracker.map.tasks.maximum appropriately on each node.
(2) In our InputFormat, we use the numSplits to tell us how many map
tasks the job's files should be split into. However, HOD does not
override the mapred.map.tasks property (nor the mapred.reduce.tasks),
while they should be set dependent on the number of available task
trackers and/or nodes in the HOD session.
Can this not be submitted via the Hadoop job's configuration ? Again,
HOD cannot do this automatically currently. But you could use the
hod.client-params to set up a client side hadoop-site.xml that would
work like this for all jobs submitted to the cluster.
Hope this helps some.
Thanks
Hemanth
