[ https://issues.apache.org/jira/browse/MAPREDUCE-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782864#action_12782864 ]
Hudson commented on MAPREDUCE-1182: ----------------------------------- Integrated in Hadoop-Mapreduce-trunk-Commit #134 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/134/]) > Reducers fail with OutOfMemoryError while copying Map outputs > ------------------------------------------------------------- > > Key: MAPREDUCE-1182 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1182 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Chandra Prakash Bhagtani > Assignee: Chandra Prakash Bhagtani > Priority: Blocker > Fix For: 0.20.2 > > Attachments: HADOOP-6357.patch, M1182-0.patch, M1182-0v20.patch, > M1182-1.patch, M1182-1v20.patch > > > Reducers fail while copying Map outputs with following exception > java.lang.OutOfMemoryError: Java heap space at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1539) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) > ,Error: > Reducer's memory usage keeps on increasing and ultimately exceeds -Xmx value > I even tried with -Xmx6.5g to each reducer but it's still failing > While looking into the reducer logs, I found that reducers were doing > shuffleInMemory every time, rather than doing shuffleOnDisk -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.