[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3681: --- Fix Version/s: 0.23.1 capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3681: - Attachment: MAPREDUCE-3681.patch I had made the updates for usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() * parent.getAbsoluteCapacity()) while testing, here is what I was running. capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Attachment: MAPREDUCE-3681.patch Trivial patch. capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Status: Patch Available (was: Open) capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Status: Open (was: Patch Available) capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Attachment: MAPREDUCE-3681.patch Findbugs caught the 'usedCapacity=0' problem... integral division! :) capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Status: Patch Available (was: Open) capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues calculate used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3681: - Priority: Critical (was: Blocker) Good catch Thomas. Thankfully this isn't super-critical because 'usedCapacity' is used nowhere other than displaying queue-stats to clients i.e. it doesn't runtime behaviour of the CapacityScheduler. I'll fix this. capacity scheduler LeafQueues calculate used capacity wrong --- Key: MAPREDUCE-3681 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Arun C Murthy Priority: Critical In the Capacity scheduler if you configure the queues to be hierarchical where you have root - parent queue - leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity. In updateResource in LeafQueue: setUsedCapacity( usedResources.getMemory() / (clusterResource.getMemory() * capacity)); I think the clusterResource.getMemory() should be something like getParentsMemory(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
