[
https://issues.apache.org/jira/browse/HADOOP-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630809#action_12630809
]
Christian Kunz commented on HADOOP-3841:
----------------------------------------
There was another instance of a reducer on a node with more than 280GB free
space distributed over 4 disks and map input of 210 GB that could not find a
disk with enough free space when it tried to create a merge file larger than 30
GB, because it included intermediate merged files into the new merge attempt
although there were enough small non-merged files that could have been merged
to bring the total number of input files below the fan factor.
> merge phase runs out of disk space
> ----------------------------------
>
> Key: HADOOP-3841
> URL: https://issues.apache.org/jira/browse/HADOOP-3841
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> We observe that reduce tasks run out of disk space during merging (after
> fetching all map output) although there would be enough space if the
> framework did not try to generate too large merge files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
