[ https://issues.apache.org/jira/browse/MAPREDUCE-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2062: ----------------------------------------- Environment: hadoop-20 with HADOOP-2141 > speculative execution is too aggressive under certain conditions > ---------------------------------------------------------------- > > Key: MAPREDUCE-2062 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2062 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker > Environment: hadoop-20 with HADOOP-2141 > Reporter: Joydeep Sen Sarma > > The function canBeSpeculated has subtle bugs that cause too much speculation > in certain cases. > - it compares the current progress of the task with the last observed mean of > all the tasks. if only one task is in question - then the progress rate > decays as time progresses (in the absence of updates) and std-dev is zero. So > a job with a single reducer or mapper is almost always speculated. > - is only a single task has reported progress - then the stddev is zero. so > other tasks may be speculated aggressively. > - several tasks take a while to report progress initially. they seem to get > speculated as soon as speculative-lag is over. the lag should be configurable > at the minimum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.