On 3/12/09 7:13 PM, "Vadim Zaliva" <[email protected]> wrote:

> The machines have 4 disk each, stripped.
> However I do not see disks being a bottleneck.

    When you stripe you automatically make every disk in the system have the
same speed as the slowest disk.  In our experiences, systems are more likely
to have a 'slow' disk than a dead one.... and detecting that is really
really hard.  In a distributed system, that multiplier effect can have
significant consequences on the whole grids performance.

    Also:

>>> 16Gb RAM each. They both run mapreduce and dfs nodes. Currently
>>> I've set up each of them to run 32 map and 8 reduce tasks.
>>> Also, HADOOP_HEAPSIZE=2048.

    IIRC, HEAPSIZE is in MB, sooo...

        32*2048=65536
        8*2048=16384

        81920MB of VM *just in heap* ...

>>> I see CPU is under utilized. If there is a guideline how I can find optimal
>>> number of tasks and memory setting for this kind of hardware.

    If you actually have all 40 task slots going, the system is likely
spending quite a bit of time paging...

    We generally use cores/2-1 for map and reduce slots.  This leaves some
cores and memory for the OS to use for monitoring, data node, etc. So 8
cores=3 maps, 3 reduces per node.  Going above that has been done for
extremely lightweight processes, but you're more likely headed for heartache
if you aren't careful.

Reply via email to