[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375783#comment-15375783 ]
Daniel Templeton commented on MAPREDUCE-6724: --------------------------------------------- Thanks for the patch, [~haibochen]! A couple comments: {code} final long totalReduceMem = 8L * 1024 * 1024 * 1024; final float shuffleInputBuf = 1.0f; final float reduceInputBuf = 1.0f; final MergeManagerImpl<Text, Text> mgr = createMergeManager( totalReduceMem, shuffleInputBuf, 0.95f, reduceInputBuf); {code} It would be better to be consistent in the parameters. Everything is a variable except for the shuffle memory limit. {code} final long totalReduceMem = 10000000000000000L; final float shuffleInputBuf = 1.0f; final float shuffleMemLimit = singleShuffleLimitConfiged / (float) totalReduceMem; final MergeManagerImpl<Text, Text> mgr = createMergeManager( totalReduceMem, shuffleInputBuf, shuffleMemLimit, 1.0f); {code} Same comment here, except with reducer input buffer. > Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve() > ----------------------------------------------------------------------------- > > Key: MAPREDUCE-6724 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 > Reporter: Haibo Chen > Assignee: Haibo Chen > Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, > mapreduce6724.003.patch > > > When shuffle is done in memory, MergeManagerImpl converts the requested size > to an int to allocate an instance of InMemoryMapOutput. This results in an > overflow if the requested size is bigger than Integer.MAX_VALUE and > eventually causes the reducer to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org