[ http://issues.apache.org/jira/browse/HADOOP-578?page=comments#action_12444642 ] Sanjay Dahiya commented on HADOOP-578: --------------------------------------
Currently a failed task moves the firstMapToTry and firstReduceToTry to next task from failed one, which will cause skipping of this task for immediate next scheduling. Although a failed task having ID less than this failed task will push the pointer back and this task may still get scheduled. We can fix this by changing - JobInProgress.failedTask() -> firstMapToTry = Math.min(firstMapToTry, tip.getIdWithinJob()) ; this will make sure that job tracker looks at first failed task in the list at any point before scheduling other tasks. This will make sure that if the job is going to fail due to certain failure of some tasks then we detect it as soon as possible. comments ? > Failed tasks should not be put at the end of the job tracker's queue > -------------------------------------------------------------------- > > Key: HADOOP-578 > URL: http://issues.apache.org/jira/browse/HADOOP-578 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Affects Versions: 0.6.2 > Reporter: Owen O'Malley > Assigned To: Sanjay Dahiya > > This functionality was basically a workaround for other problems, which have > been fixed. The impact of putting them at the end of the work queue is that > if you have 100,000 maps, and the Mapper fails deterministically, you'll run > 300,000+ attempts before your job is killed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
