[jira] [Commented] (MAPREDUCE-2684) Job Tracker can starve reduces with very large input.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064923#comment-13064923 ] Aaron T. Myers commented on MAPREDUCE-2684: --- Hey Robert, is this not a duplicate of https://issues.apache.org/jira/browse/MAPREDUCE-2324 ? > Job Tracker can starve reduces with very large input. > - > > Key: MAPREDUCE-2684 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2684 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.204.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > If mapreduce.reduce.input.limit is mis-configured or if a cluster is just > running low on disk space in general then reduces with large a input may > never get scheduled causing the Job to never fail and never succeed, just > starve until the job is killed. > The JobInProgess tries to guess at the size of the input to all reducers in a > job. If the size is over mapreduce.reduce.input.limit then the job is > killed. If it is not then findNewReduceTask() checks to see if the estimated > size is too big to fit on the node currently looking for work. If it is not > then it will let some other task have a chance at the slot. > The idea is to keep track of how often it happens that a Reduce Slot is > rejected because of the lack of space vs how often it succeeds and then guess > if the reduce tasks will ever be scheduled. > So I would like some feedback on this. > 1) How should we guess. Someone who found the bug here suggested P1 + (P2 * > S), where S is the number of successful assignments. Possibly P1 = 20 and P2 > = 2.0. I am not really sure. > 2) What should we do when we guess that it will never get a slot? Should we > fail the job or do we say, even though it might fail, well lets just schedule > the it and see if it really will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2684) Job Tracker can starve reduces with very large input.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065248#comment-13065248 ] Robert Joseph Evans commented on MAPREDUCE-2684: You are correct I will duplicate this one to that one, because that one was filed first. > Job Tracker can starve reduces with very large input. > - > > Key: MAPREDUCE-2684 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2684 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.204.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > If mapreduce.reduce.input.limit is mis-configured or if a cluster is just > running low on disk space in general then reduces with large a input may > never get scheduled causing the Job to never fail and never succeed, just > starve until the job is killed. > The JobInProgess tries to guess at the size of the input to all reducers in a > job. If the size is over mapreduce.reduce.input.limit then the job is > killed. If it is not then findNewReduceTask() checks to see if the estimated > size is too big to fit on the node currently looking for work. If it is not > then it will let some other task have a chance at the slot. > The idea is to keep track of how often it happens that a Reduce Slot is > rejected because of the lack of space vs how often it succeeds and then guess > if the reduce tasks will ever be scheduled. > So I would like some feedback on this. > 1) How should we guess. Someone who found the bug here suggested P1 + (P2 * > S), where S is the number of successful assignments. Possibly P1 = 20 and P2 > = 2.0. I am not really sure. > 2) What should we do when we guess that it will never get a slot? Should we > fail the job or do we say, even though it might fail, well lets just schedule > the it and see if it really will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira