Xeon vs. Opteron is likely not going to be a major factor. More important
than this is the number of disks you have per machine. Task performance is
proportional to both the number of CPUs and the number of disks.

You are probably using way too many tasks. Adding more tasks/node isn't
necessarily going to increase utilization if they're waiting on data from
the disks; you'll just increase the IO pressure and probably make things
seek more. You've configured up to 40 simultaneous tasks/node, to run on 8
cores. I'd lower that to maybe 8 maps and 6 reduces. You might get better
usage since they don't compete for IO as much. The correct value for you
might lie somewhere in between, too, depending on your workload.

How many hdds does your machine have? You should stripe dfs.data.dir and
mapred.local.dir across all of them. Having a 2:1 core:disk (And thus a 2:1
map task:disk) ratio is probably a good starting point, so for eight cores,
you should have at least 4 disks.

- Aaron

On Wed, Mar 11, 2009 at 10:15 AM, Vadim Zaliva <[email protected]> wrote:

> Hi!
>
> I have a question about fine-tunining hadoop performance on 8-core
> machines.
> I have 2 machines I am testing. One is 8-core Xeon and another is 8-core
> Opteron. 16Gb RAM each. They both run mapreduce and dfs nodes. Currently
> I've set up each of them to run 32 map and 8 reduce tasks.
> Also, HADOOP_HEAPSIZE=2048.
>
> I see CPU is under utilized. If there is a guideline how I can find optimal
> number of tasks and memory setting for this kind of hardware.
>
> Also, since we going to my more machines like this, I need to decided
> whenever buy Xeons or Opterons. Any advise on that?
>
> Sincerely,
> Vadim
>
> P.S. I am using Hadoop 19 and java version "1.6.0_12":
> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
>

Reply via email to