Hi,
First, it sounds like you have 2 6 core CPUs giving you 12 cores not 24.
Even though the OS reports 24 cores that's the hyper threading and not real
cores.
This becomes an issue with respect to tuning.
To answer your question ...
You have a single 1TB HD. That's going to be a major bottleneck in terms of
performance. You usually want to have at least 1 drive per core. With a 12
core box that's 12 spindles.
How large is your hadoop job's jar? This gets pushed around to all of the
nodes.
Bigger jars take longer to process and handle.
Having said that, the start up time isn't out of whack.
It depends on what job you're launching and what you are doing within the job.
Remember that the tasks have to report back to the JT.
Do you have Ganglia up and running?
You will probably see a high load on the CPUs and then a lot of Wait IOs.
HTH
-Mike
On Mar 20, 2012, at 5:40 AM, praveenesh kumar wrote:
I have 10 node cluster ( around 24 CPUs, 48 GB RAM, 1 TB HDD, 10 GB
ethernet connection)
After triggering any MR job, its taking like 3-5 seconds to launch ( I mean
the time when I can see any MR job completion % on the screen).
I know internally its trying to launch the job,intialize mappers, loading
data etc.
What I want to know - Is it a default/desired/expected hadoop behavior or
there are ways in which I can decrease this startup time ?
Also I feel like my hadoop jobs should run faster, but I am still not able
to make it as fast as it should be according to me ?
I did some tunning also, following are the parameters I am playing around
these days but still I feel there are something missing that I can still
use:
dfs.block.size:
mapred.compress.map.output
mapred.map/reduce.tasks.speculative.execution
mapred.tasktracker.map/reduce.tasks.maximum:
mapred.child.java.opts
io.sort.mb:
io.sort.factor:
mapred.reduce.parallel.copies:
mapred.job.reuse.jvm.num.tasks:
Thanks,
Praveenesh
