The machines have 4 disk each, stripped. However I do not see disks being a bottleneck. Monitoring system activity shows that CPU is utilized 2-70%, disk usage is moderate, while network activity seems to be quite high. In this particular cluster we have 6 machines and replication factor is 2. I was wondering if increasing replication factor would help, so there is a better chance that data block is available locally.
Sincerely, Vadim On Thu, Mar 12, 2009 at 13:27, Aaron Kimball <[email protected]> wrote: > Xeon vs. Opteron is likely not going to be a major factor. More important > than this is the number of disks you have per machine. Task performance is > proportional to both the number of CPUs and the number of disks. > > You are probably using way too many tasks. Adding more tasks/node isn't > necessarily going to increase utilization if they're waiting on data from > the disks; you'll just increase the IO pressure and probably make things > seek more. You've configured up to 40 simultaneous tasks/node, to run on 8 > cores. I'd lower that to maybe 8 maps and 6 reduces. You might get better > usage since they don't compete for IO as much. The correct value for you > might lie somewhere in between, too, depending on your workload. > > How many hdds does your machine have? You should stripe dfs.data.dir and > mapred.local.dir across all of them. Having a 2:1 core:disk (And thus a 2:1 > map task:disk) ratio is probably a good starting point, so for eight cores, > you should have at least 4 disks. > > - Aaron > > On Wed, Mar 11, 2009 at 10:15 AM, Vadim Zaliva <[email protected]> wrote: > >> Hi! >> >> I have a question about fine-tunining hadoop performance on 8-core >> machines. >> I have 2 machines I am testing. One is 8-core Xeon and another is 8-core >> Opteron. 16Gb RAM each. They both run mapreduce and dfs nodes. Currently >> I've set up each of them to run 32 map and 8 reduce tasks. >> Also, HADOOP_HEAPSIZE=2048. >> >> I see CPU is under utilized. If there is a guideline how I can find optimal >> number of tasks and memory setting for this kind of hardware. >> >> Also, since we going to my more machines like this, I need to decided >> whenever buy Xeons or Opterons. Any advise on that? >> >> Sincerely, >> Vadim >> >> P.S. I am using Hadoop 19 and java version "1.6.0_12": >> Java(TM) SE Runtime Environment (build 1.6.0_12-b04) >> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode) >> >
