[
https://issues.apache.org/jira/browse/HADOOP-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651787#action_12651787
]
Chris Douglas commented on HADOOP-4714:
---------------------------------------
bq. Although this would solve the issue for our particular case, I can imagine
a situation (e.g. single reducer with highly aggregated huge records) where
this would not help, i.e. the time component needs to be factored into the
progress reporting. Progress should always be reported at smaller intervals
than the timeout which is configurable and could be a small number.
Fair point. Still, it's far from the only profile that makes unobserved
progress and the current approach isn't guaranteed to work in this hypothetical
case, either. Balancing performance against accuracy in the current model
admits the possibility of spurious timeouts by definition. The heuristics we
use- every N records out of the merge, every partition, etc.- are, as you point
out, not guaranteed to fit each job's profile, but they're usually good enough
without being too expensive. Continuing to refine them is a stop-gap,
admittedly.
The TaskTracker could try to look for external signs of progress in tasks it
suspects are stuck (e.g. spills/merges generated between heartbeats), but that
mistakes MapTask side-effects for task health. Adding a thread to poll for
progress within the task adds modest benefits, but the cost in complexity (and
likely performance) is discouraging. It also separates the checks for progress
from the code effecting it, making it harder to maintain. While it's possible
to imagine an adaptive system that would sample the frequency of status updates
generated from each component and tune each threshold to the particular job,
that's a long, long way from what is currently in place.
For the scope of this JIRA, I think it's sufficient to flag progress after each
partition. Doing more reporting out of the merge would be OK, but I see no
reason to enshrine the N records/progress update heuristic.
> map tasks timing out during merge phase
> ---------------------------------------
>
> Key: HADOOP-4714
> URL: https://issues.apache.org/jira/browse/HADOOP-4714
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.1
> Reporter: Christian Kunz
> Assignee: Jothi Padmanabhan
> Attachments: hadoop-4714.patch
>
>
> With compression of transient data turned on some parts of the merge phase
> seem to not report progress enough.
> We see a lot of task failures during the merge phase, most of them timing out
> (even with a 20 min timeout)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
