[
https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15375783#comment-15375783
]
Daniel Templeton commented on MAPREDUCE-6724:
---------------------------------------------
Thanks for the patch, [~haibochen]! A couple comments:
{code}
final long totalReduceMem = 8L * 1024 * 1024 * 1024;
final float shuffleInputBuf = 1.0f;
final float reduceInputBuf = 1.0f;
final MergeManagerImpl<Text, Text> mgr = createMergeManager(
totalReduceMem, shuffleInputBuf, 0.95f, reduceInputBuf);
{code}
It would be better to be consistent in the parameters. Everything is a
variable except for the shuffle memory limit.
{code}
final long totalReduceMem = 10000000000000000L;
final float shuffleInputBuf = 1.0f;
final float shuffleMemLimit =
singleShuffleLimitConfiged / (float) totalReduceMem;
final MergeManagerImpl<Text, Text> mgr = createMergeManager(
totalReduceMem, shuffleInputBuf, shuffleMemLimit, 1.0f);
{code}
Same comment here, except with reducer input buffer.
> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-6724
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch,
> mapreduce6724.003.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size
> to an int to allocate an instance of InMemoryMapOutput. This results in an
> overflow if the requested size is bigger than Integer.MAX_VALUE and
> eventually causes the reducer to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
