The machines have 4 disk each, stripped.
However I do not see disks being a bottleneck. Monitoring system activity
shows that CPU is utilized 2-70%, disk usage is moderate, while network
activity seems to be quite high. In this particular cluster we have 6 machines
and replication factor is 2. I was wondering if increasing replication
factor would
help, so there is a better chance that data block is available locally.

Sincerely,
Vadim


On Thu, Mar 12, 2009 at 13:27, Aaron Kimball <[email protected]> wrote:
> Xeon vs. Opteron is likely not going to be a major factor. More important
> than this is the number of disks you have per machine. Task performance is
> proportional to both the number of CPUs and the number of disks.
>
> You are probably using way too many tasks. Adding more tasks/node isn't
> necessarily going to increase utilization if they're waiting on data from
> the disks; you'll just increase the IO pressure and probably make things
> seek more. You've configured up to 40 simultaneous tasks/node, to run on 8
> cores. I'd lower that to maybe 8 maps and 6 reduces. You might get better
> usage since they don't compete for IO as much. The correct value for you
> might lie somewhere in between, too, depending on your workload.
>
> How many hdds does your machine have? You should stripe dfs.data.dir and
> mapred.local.dir across all of them. Having a 2:1 core:disk (And thus a 2:1
> map task:disk) ratio is probably a good starting point, so for eight cores,
> you should have at least 4 disks.
>
> - Aaron
>
> On Wed, Mar 11, 2009 at 10:15 AM, Vadim Zaliva <[email protected]> wrote:
>
>> Hi!
>>
>> I have a question about fine-tunining hadoop performance on 8-core
>> machines.
>> I have 2 machines I am testing. One is 8-core Xeon and another is 8-core
>> Opteron. 16Gb RAM each. They both run mapreduce and dfs nodes. Currently
>> I've set up each of them to run 32 map and 8 reduce tasks.
>> Also, HADOOP_HEAPSIZE=2048.
>>
>> I see CPU is under utilized. If there is a guideline how I can find optimal
>> number of tasks and memory setting for this kind of hardware.
>>
>> Also, since we going to my more machines like this, I need to decided
>> whenever buy Xeons or Opterons. Any advise on that?
>>
>> Sincerely,
>> Vadim
>>
>> P.S. I am using Hadoop 19 and java version "1.6.0_12":
>> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
>> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
>>
>

Reply via email to