For a simple test, set the replication on your entire cluster to 6 hadoop
dfs -setRep  -R  -w 6 /

This will triple your disk usage and probably take a while, but then you are
guaranteed that all data is local.

You can also get a rough idea from the Job Counters, 'Data-local map tasks'
total field, of the data local map tasks.



-setrep [-R] [-w] <rep> <path/file>:  Set the replication level of a file.
        The -R flag requests a recursive change of replication level
        for an entire tree.


On Thu, Mar 12, 2009 at 7:13 PM, Vadim Zaliva <[email protected]> wrote:

> The machines have 4 disk each, stripped.
> However I do not see disks being a bottleneck. Monitoring system activity
> shows that CPU is utilized 2-70%, disk usage is moderate, while network
> activity seems to be quite high. In this particular cluster we have 6
> machines
> and replication factor is 2. I was wondering if increasing replication
> factor would
> help, so there is a better chance that data block is available locally.
>
> Sincerely,
> Vadim
>
>
> On Thu, Mar 12, 2009 at 13:27, Aaron Kimball <[email protected]> wrote:
> > Xeon vs. Opteron is likely not going to be a major factor. More important
> > than this is the number of disks you have per machine. Task performance
> is
> > proportional to both the number of CPUs and the number of disks.
> >
> > You are probably using way too many tasks. Adding more tasks/node isn't
> > necessarily going to increase utilization if they're waiting on data from
> > the disks; you'll just increase the IO pressure and probably make things
> > seek more. You've configured up to 40 simultaneous tasks/node, to run on
> 8
> > cores. I'd lower that to maybe 8 maps and 6 reduces. You might get better
> > usage since they don't compete for IO as much. The correct value for you
> > might lie somewhere in between, too, depending on your workload.
> >
> > How many hdds does your machine have? You should stripe dfs.data.dir and
> > mapred.local.dir across all of them. Having a 2:1 core:disk (And thus a
> 2:1
> > map task:disk) ratio is probably a good starting point, so for eight
> cores,
> > you should have at least 4 disks.
> >
> > - Aaron
> >
> > On Wed, Mar 11, 2009 at 10:15 AM, Vadim Zaliva <[email protected]>
> wrote:
> >
> >> Hi!
> >>
> >> I have a question about fine-tunining hadoop performance on 8-core
> >> machines.
> >> I have 2 machines I am testing. One is 8-core Xeon and another is 8-core
> >> Opteron. 16Gb RAM each. They both run mapreduce and dfs nodes. Currently
> >> I've set up each of them to run 32 map and 8 reduce tasks.
> >> Also, HADOOP_HEAPSIZE=2048.
> >>
> >> I see CPU is under utilized. If there is a guideline how I can find
> optimal
> >> number of tasks and memory setting for this kind of hardware.
> >>
> >> Also, since we going to my more machines like this, I need to decided
> >> whenever buy Xeons or Opterons. Any advise on that?
> >>
> >> Sincerely,
> >> Vadim
> >>
> >> P.S. I am using Hadoop 19 and java version "1.6.0_12":
> >> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
> >> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
> >>
> >
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

Reply via email to