[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6610: -- Attachment: YARN-6610.YARN-3926.007.patch Attaching same patch shared by [~leftnoteasy] fixing an unused import and method name change. +1 on this patch. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch, > YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.007.patch, > YARN-6610.YARN-3926.perf.patch, YARN-6610.YARN-3926.perf-rebased.patch, > YARN-6610.YARN-3926.perf-wangda-001.patch, > YARN-6610.YARN-3926.perf-wangda-002.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6610: - Attachment: YARN-6610.YARN-3926.perf-wangda-002.patch Updated patch (wangda-002) > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch, > YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch, > YARN-6610.YARN-3926.perf-rebased.patch, > YARN-6610.YARN-3926.perf-wangda-001.patch, > YARN-6610.YARN-3926.perf-wangda-002.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6610: - Attachment: YARN-6610.YARN-3926.perf-wangda-001.patch Investigated more, it looks like original perf patch has some typos inside {{calculateShares2}}, I tried to run all CapacityScheduler tests with the old perf patch, it causes many failures. With attached patch {{YARN-6610.YARN-3926.perf-wangda-001.patch}}, it has 2 failures, but looks unrelated. (Ran unit test w/o the patch, it fails as well). [~templedf], it looks like existing tests in TestResourceCalculator are not enough to capture the issue, I tried to add some (testCompare2) but it looks not enough as well. If you have any cycles, could you help to add more unit tests? I also tried to run testUserLimitThroughput again, the latest patch shows very similar result. Since this is blocker of YARN-3926 merge, and it doesn't show performance regression, I plan to commit this patch to branch by end of today if no opposite opinions. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch, > YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch, > YARN-6610.YARN-3926.perf-rebased.patch, > YARN-6610.YARN-3926.perf-wangda-001.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6610: - Attachment: YARN-6610.YARN-3926.006-rebased.patch YARN-6610.YARN-3926.perf-rebased.patch Attached rebased patch (with rebased suffix) > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch, > YARN-6610.YARN-3926.006-rebased.patch, YARN-6610.YARN-3926.perf.patch, > YARN-6610.YARN-3926.perf-rebased.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.perf.patch Here's a patch that adds a special case for # res == 2. [~sunilg], any chance you can run it through SLS and see what the difference is? > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch, > YARN-6610.YARN-3926.perf.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.006.patch Added unit tests and fixed an error. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.005.patch I added a handler for the exception. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch, > YARN-6610.YARN-3926.005.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.004.patch Refactored a little to make it cleaner. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.003.patch Realized that I had the sorts backwards. New patch attached. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.002.patch Now that YARN-6788 is in, here's a fresh patch that is significantly better optimized. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6610: -- Issue Type: Sub-task (was: Bug) Parent: YARN-3926 > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.001.patch Here's a first pass at making the compare make sense again. Thoughts? The patch still needs tests. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org