[jira] [Commented] (MAPREDUCE-4867) reduces tasks won't start in certain circumstances
[ https://issues.apache.org/jira/browse/MAPREDUCE-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527994#comment-13527994 ] Vincent Behar commented on MAPREDUCE-4867: -- yes it is a duplicate of MAPREDUCE-2129 (sorry I didn't find it) The fix has been applied to branch-1 and branch-1.1, but not branch-1.0. Merging r1358233 (from branch-1) in branch-1.0 should be enough. Thanks > reduces tasks won't start in certain circumstances > --- > > Key: MAPREDUCE-4867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: scheduler >Affects Versions: 1.0.4 >Reporter: Vincent Behar > > Reduce tasks start are conditioned by the value of > "mapred.reduce.slowstart.completed.maps". However, if the number of completed > map tasks never reached the configured value (for example because > "mapred.max.map.failures.percent" has been set to a high value, to permit a > job to have a lot of failed tasks), then the reduce tasks won't start. > The job is still running, all map tasks are finished (either successful or > not), and all reduce tasks are still pending. The only thing one can do is to > kill the job. > There are 2 things that could be done : > - document the relation between "mapred.max.map.failures.percent" and > "mapred.reduce.slowstart.completed.maps" : we can say that the rule to follow > if you want to be sure that your reduce tasks will start is : > "mapred.reduce.slowstart.completed.maps * 100 < 100 - > mapred.max.map.failures.percent" > - fix JobInProgress.scheduleReduces() to return true if all map tasks are > finished -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4867) reduces tasks won't start in certain circumstances
Vincent Behar created MAPREDUCE-4867: Summary: reduces tasks won't start in certain circumstances Key: MAPREDUCE-4867 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4867 Project: Hadoop Map/Reduce Issue Type: Bug Components: scheduler Affects Versions: 1.0.4 Reporter: Vincent Behar Reduce tasks start are conditioned by the value of "mapred.reduce.slowstart.completed.maps". However, if the number of completed map tasks never reached the configured value (for example because "mapred.max.map.failures.percent" has been set to a high value, to permit a job to have a lot of failed tasks), then the reduce tasks won't start. The job is still running, all map tasks are finished (either successful or not), and all reduce tasks are still pending. The only thing one can do is to kill the job. There are 2 things that could be done : - document the relation between "mapred.max.map.failures.percent" and "mapred.reduce.slowstart.completed.maps" : we can say that the rule to follow if you want to be sure that your reduce tasks will start is : "mapred.reduce.slowstart.completed.maps * 100 < 100 - mapred.max.map.failures.percent" - fix JobInProgress.scheduleReduces() to return true if all map tasks are finished -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4849) TaskSelector not used in FairScheduler
Vincent Behar created MAPREDUCE-4849: Summary: TaskSelector not used in FairScheduler Key: MAPREDUCE-4849 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4849 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 1.1.1, 1.0.4 Reporter: Vincent Behar The documentation (http://hadoop.apache.org/docs/r1.0.4/fair_scheduler.html) describes the mapred.fairscheduler.taskselector parameter as an "extension point", but while the FairScheduler does instantiate the custom TaskSelector provided this way, it does not call any of its methods (obtainNewMapTask, obtainNewReduceTask, neededSpeculativeMaps or neededSpeculativeReduces). We should either update the FairScheduler to use the TaskSelector when scheduling a task, or completely remove the TaskSelector and update the documentation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
