Xeon vs. Opteron is likely not going to be a major factor. More important than this is the number of disks you have per machine. Task performance is proportional to both the number of CPUs and the number of disks.
You are probably using way too many tasks. Adding more tasks/node isn't necessarily going to increase utilization if they're waiting on data from the disks; you'll just increase the IO pressure and probably make things seek more. You've configured up to 40 simultaneous tasks/node, to run on 8 cores. I'd lower that to maybe 8 maps and 6 reduces. You might get better usage since they don't compete for IO as much. The correct value for you might lie somewhere in between, too, depending on your workload. How many hdds does your machine have? You should stripe dfs.data.dir and mapred.local.dir across all of them. Having a 2:1 core:disk (And thus a 2:1 map task:disk) ratio is probably a good starting point, so for eight cores, you should have at least 4 disks. - Aaron On Wed, Mar 11, 2009 at 10:15 AM, Vadim Zaliva <[email protected]> wrote: > Hi! > > I have a question about fine-tunining hadoop performance on 8-core > machines. > I have 2 machines I am testing. One is 8-core Xeon and another is 8-core > Opteron. 16Gb RAM each. They both run mapreduce and dfs nodes. Currently > I've set up each of them to run 32 map and 8 reduce tasks. > Also, HADOOP_HEAPSIZE=2048. > > I see CPU is under utilized. If there is a guideline how I can find optimal > number of tasks and memory setting for this kind of hardware. > > Also, since we going to my more machines like this, I need to decided > whenever buy Xeons or Opterons. Any advise on that? > > Sincerely, > Vadim > > P.S. I am using Hadoop 19 and java version "1.6.0_12": > Java(TM) SE Runtime Environment (build 1.6.0_12-b04) > Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode) >
