On May 17, 2011, at 3:11 PM, Mark question wrote:
> So what other memory consumption tools do you suggest? I don't want to do it
> manually and dump statistics into file because IO will affect performance
> too.
We watch memory with Ganglia. We also tune our systems such that a
task will only take X amount. In other words, given an 8gb RAM:
1gb for the OS
1gb for the TT and DN
6gb for all tasks
if we assume each task will take max 1gb, then we end up with 3 maps
and 3 reducers.
Keep in mind that the mem consumed is more than just JVM heap size.
