[
https://issues.apache.org/jira/browse/MAPREDUCE-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989944#comment-13989944
]
Karthik Kambatla commented on MAPREDUCE-5877:
---------------------------------------------
This issue is reported on MAPREDUCE-2217, and we have also seen this in some
CDH users. After investigating this issue in the context of MAPREDUCE-5110 and
otherwise, I strongly believe that MAPREDUCE-2217 was the wrong way of handling
things. IMO, it is more a hack than a good solution.
A better approach would be for the TT to monitor the progress of tasks
including those that haven't been launched yet (per-job config:
mapred.task.launch.timeout) and kill those that are past the timeout. The JT
realizes this and launches another attempt. In case the TT itself expires, the
task yet to be launched also gets expired along with other running tasks. We
have tested this fix on production clusters and it works significantly better
than the one on MAPREDUCE-2217.
> Inconsistency between JT/TT for tasks taking a long time to launch
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-5877
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5877
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 1.2.1
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Priority: Critical
>
> For the tasks that take too long to launch (for genuine reasons like large
> distributed caches), JT expires the task. Depending on whether job recovery
> is enabled and the JT's restart state, another attempt is launched or not
> even when the JT is not restarted. The status of the attempt changes to
> "Error launching task". Meanwhile, the TT is not informed of this task expiry
> and eventually launches the task.
> To avoid this weird behavior, one can bump up the
> mapred.tasktracker.expiry.interval, but leading to long TT failure discovery
> times.
> We should have a per-job timeout for task launches/ heartbeat and JT/TT
> should be consistent in what they say.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
