[
https://issues.apache.org/jira/browse/MAPREDUCE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794920#comment-13794920
]
zhaoyunjiong commented on MAPREDUCE-5541:
-----------------------------------------
I have run some "Random sleep job" to test.
It works fine.
The previous implement use "currentTime - startTime >= SPECULATIVE_LAG" to
decide whether need speculative tasks, but the "startTime" is set when
JobTracker init job.
In reality, tasks won't start at same time due to resource limits, so if there
are part of tasks finished, it may cause later running tasks need speculative
task even it is just start, that will waste lots of resource.
> Improved algorithm for whether need speculative task
> ----------------------------------------------------
>
> Key: MAPREDUCE-5541
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5541
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1
> Affects Versions: 1.2.1
> Reporter: zhaoyunjiong
> Assignee: zhaoyunjiong
> Fix For: 1.2.2
>
> Attachments: MAPREDUCE-5541-branch-1.2.patch,
> MAPREDUCE-5541-branch-1.2.patch
>
>
> Most of time, tasks won't start running at same time.
> In this case hasSpeculativeTask in TaskInProgress not working very well.
> Some times, some tasks just start running, and scheduler already decide it
> need speculative task to run.
> And this waste a lot of resource.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
