[
https://issues.apache.org/jira/browse/HADOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated HADOOP-3140:
----------------------------------
Description:
In most cases, map tasks do not write to dfs.
Thus, when they complete, they should not be put into commit_pending queue at
all.
This will improve the task promotion significantly.
was:
In most cases, map tasks do not write to dfs.
Thus, when they complete, they should not be put into commit_pending queue at
all.
This will improve the task promotion significantly.
Summary: JobTracker should not try to promote a (map) task if it does
not write to DFS at all (was: JobTracker should not try to promote a (map)
task if it dis not write to DFS at all)
I think that the tasks should include a boolean in the done message to the task
tracker that says if they have output to promote. (And it should delete
everything in the case of failure, locally.) This is just an optimization. The
framework (TaskTracker.Child.main) would look in the work output directory and
set true if there is anything to promote. The TT would then set the state to
commit-pending or success according to the flag value.
> JobTracker should not try to promote a (map) task if it does not write to DFS
> at all
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3140
> URL: https://issues.apache.org/jira/browse/HADOOP-3140
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> In most cases, map tasks do not write to dfs.
> Thus, when they complete, they should not be put into commit_pending queue at
> all.
> This will improve the task promotion significantly.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
