I'm seeing this error when a job runs: Shuffling 35338524 bytes (35338524 raw bytes) into RAM from attempt_201001051549_0036_m_000003_0 Map output copy failure: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1198)
I originally had mapred.child.java.opts set to 200M. If I boost this up to 512M the error goes away. I'm trying to understand whats going on though. Can anyone explain? Also are there any other parameters that
I should be tweaking to help with this? thank you very much, M
