[
https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651807#action_12651807
]
Jothi Padmanabhan commented on HADOOP-4714:
-------------------------------------------
Had an offline discussion with Chris. It looks like that we could do the
following to address the issue without complicating the code much
* Change Merger.writeFile
{code}
if ((++recordCtr % PROGRESS_BAR) == 0) {
progressable.progress();
}
{code}
to
{code}
if ((recordCtr++ % PROGRESS_BAR) == 0) {
progressable.progress();
}
{code}
so that progress is sent at the beginning for each reducer
* Make the threshold (the record counts after which progress is sent,
PROGRESS_BAR in the current code) configurable, by adding that as a parameter
in hadoop_default.xml. The existing 10,000 would be the default. That way,
users can easily arrive at the optimal number for their specific
application/typical data set, if they so wish.
Agreed that this will not address all the use cases, but should be sufficient
for most. Thoughts?
> map tasks timing out during merge phase
> ---------------------------------------
>
> Key: HADOOP-4714
> URL: https://issues.apache.org/jira/browse/HADOOP-4714
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.1
> Reporter: Christian Kunz
> Assignee: Jothi Padmanabhan
> Attachments: hadoop-4714.patch
>
>
> With compression of transient data turned on some parts of the merge phase
> seem to not report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out
> (even with a 20 min timeout)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
