[
https://issues.apache.org/jira/browse/MAPREDUCE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650852#comment-13650852
]
Jason Lowe commented on MAPREDUCE-5168:
---------------------------------------
Thanks for the reviews, Karthik and Vinod.
This is a case where 0.23 and branch-2/trunk have diverged. Before
MAPREDUCE-4808 the merge threads took a list of MapOutput objects to merge, and
these objects could linger around for quite some time waiting to be merged.
Therefore this patch definitely helps on 0.23. After MAPREDUCE-4808 the
MapOutput objects are relatively short-lived, as they only exist for the
duration of the file transfer. I doubt moving the creation of the
output-stream from the constructor to the shuffle method will help much, since
the Fetcher thread moves straight from the MapOutput constructor to the shuffle
phase without delay.
So this is an important fix for 0.23 and should be harmless for trunk/branch-2.
It's mostly preventative for the latter in case someone refactors the code and
MapOutput objects could linger again.
> Reducer can OOM during shuffle because on-disk output stream not released
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-5168
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5168
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.7, 2.0.5-beta
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: MAPREDUCE-5168-branch-0.23.patch, MAPREDUCE-5168.patch
>
>
> If a reducer needs to shuffle a map output to disk, it opens an output stream
> and writes the data to disk. However it does not release the reference to
> the output stream within the MapOutput, and the output stream can have a 128K
> buffer attached to it. If enough of these on-disk outputs are queued up
> waiting to be merged, it can cause the reducer to OOM during the shuffle
> phase. In one case I saw there were 1200 on-disk outputs queued up to be
> merged, leading to an extra 150MB of pressure on the heap due to the output
> stream buffers that were no longer necessary.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
