[
https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347379#comment-15347379
]
Gera Shegalov commented on MAPREDUCE-6724:
------------------------------------------
Thanks for the patch [~haibochen]. a couple of suggestions after skimming the
patch.
- It would be helpful to have a diagnostic LOG.info message that the single
shuffle limit is less than would be per
mapreduce.reduce.shuffle.memory.limit.percent config.
- we can have a much smaller patch.
-- instead of doing refactoring to make maxSingleShuffleLimit visible simply
observe the result of mapOutput.getDescription similar to
testZeroShuffleMemoryLimitPercent
-- similar tests did not have to introduce extra mock classes, and we can do
without it in this patch as well
-- the easiest seems to factor out some common logic from testLargeMemoryLimits
into a private method that can accept targetSingleShuffleLimit as a parameter.
Then you can do:
{code}
conf.setFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT,
((float)targetSingleShuffleLimit /
conf.getLong(MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES)));
{code}
We can assert DISK for targetSingleShuffleLimit > Integer.MAX_VALUE and MEMORY
otherwise.
> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-6724
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size
> to an int to allocate an instance of InMemoryMapOutput. This results in an
> overflow if the requested size is bigger than Integer.MAX_VALUE and
> eventually causes the reducer to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
