[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509866#comment-14509866 ]
Jason Lowe commented on MAPREDUCE-5649: --------------------------------------- In general looks OK to me. Couple of nits: We're taking a transient computation and storing it long-term apparently only for unit testing purposes. I think we can unit test it in a way that doesn't require long-term storage. We could allow createInMemorySegments to be overridden in a derived test class so we could spy on the parameters to verify the value comes out as expected. This runtime exception message is not going to be very user-friendly. I realize it did not originate with this patch, but since we're touching that code it would be nice to clean it up a bit: {code} + if (maxRedPer > 1.0 || maxRedPer < 0.0) { + throw new RuntimeException(MRJobConfig.REDUCE_INPUT_BUFFER_PERCENT + + maxRedPer); + } {code} > Reduce cannot use more than 2G memory for the final merge > ---------------------------------------------------------- > > Key: MAPREDUCE-5649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Reporter: stanley shi > Assignee: Gera Shegalov > Attachments: MAPREDUCE-5649.001.patch > > > In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in > the finalMerge method: > int maxInMemReduce = (int)Math.min( > Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); > > This means no matter how much memory user has, reducer will not retain more > than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)