[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347379#comment-15347379 ]
Gera Shegalov commented on MAPREDUCE-6724: ------------------------------------------ Thanks for the patch [~haibochen]. a couple of suggestions after skimming the patch. - It would be helpful to have a diagnostic LOG.info message that the single shuffle limit is less than would be per mapreduce.reduce.shuffle.memory.limit.percent config. - we can have a much smaller patch. -- instead of doing refactoring to make maxSingleShuffleLimit visible simply observe the result of mapOutput.getDescription similar to testZeroShuffleMemoryLimitPercent -- similar tests did not have to introduce extra mock classes, and we can do without it in this patch as well -- the easiest seems to factor out some common logic from testLargeMemoryLimits into a private method that can accept targetSingleShuffleLimit as a parameter. Then you can do: {code} conf.setFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT, ((float)targetSingleShuffleLimit / conf.getLong(MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES))); {code} We can assert DISK for targetSingleShuffleLimit > Integer.MAX_VALUE and MEMORY otherwise. > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > ----------------------------------------------------------------------------- > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Reporter: Haibo Chen > Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org