> When you stripe you automatically make every disk in the system have the > same speed as the slowest disk. In our experiences, systems are more likely > to have a 'slow' disk than a dead one.... and detecting that is really > really hard. In a distributed system, that multiplier effect can have > significant consequences on the whole grids performance.
All disk are the same, so there is no speed difference. Stripping can speed up disk reads (4x more heads to read at the same time). > If you actually have all 40 task slots going, the system is likely > spending quite a bit of time paging... interestingly, the swap usage is 0. Looks like my tasks are not utilizing all alloted heap. > We generally use cores/2-1 for map and reduce slots. This leaves some > cores and memory for the OS to use for monitoring, data node, etc. So 8 > cores=3 maps, 3 reduces per node. Going above that has been done for > extremely lightweight processes, but you're more likely headed for heartache > if you aren't careful. I will try to reduce number of tasks per your recommendation to see if it would make any significant difference. Thanks you for all suggestions. Vadim
