[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312385#comment-16312385 ] Zhao Yi Ming commented on YARN-7684: [~sunilg] Sorry for trouble you again! You said I can create a 2.7 patch include the YARN-4484 fix, but I am not sure how to do it. Because in the HowToContribute doc I saw, it said "If you are unsure of the target branch then trunk is usually the best choice. Committers will usually merge the patch to downstream branches." I guess the YARN-4484 fix already include in the trunk branch, so only the committers can merge the patch to 2.7 branch, right? Thanks! > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.8.0 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311006#comment-16311006 ] Sunil G commented on YARN-7684: --- Thanks [~zhaoyim] I re-opened to close this as Wont fix. and you could provide 2.7 patch in YARN-4484 to include in next release. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.8.0 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310981#comment-16310981 ] ASF GitHub Bot commented on YARN-7684: -- Github user zhaoyim closed the pull request at: https://github.com/apache/hadoop/pull/320 > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.8.0 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310979#comment-16310979 ] Zhao Yi Ming commented on YARN-7684: [~sunilg] Thanks for clarify! If so, I will close this issue. Thanks again for you time! > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.7.3 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307727#comment-16307727 ] Sunil G commented on YARN-7684: --- I will say that clicking on NodeLabels link on nodes count wont redirect to show per label resource. Its not designed in that way. We were planning to show total cluster resource. And in Node Labels page, we can see per label whats the total resource on last column. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.7.3 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307653#comment-16307653 ] Zhao Yi Ming commented on YARN-7684: [~sunilg] I porting the YARN-4484 patch back to 2.7.3 and test the fix in my 2.7.3 cluster, The total mem and cpus are show correct. And after submit jobs, they are still show correct. But in my understand view total resource per label is not correct. I paste my test results [^TestResultsAfterPortingYARN-4484.pdf] in the attachment. Could you have a look? Thanks a lot! > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.7.3 > > Attachments: TestResultsAfterPortingYARN-4484.pdf, > YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306359#comment-16306359 ] Zhao Yi Ming commented on YARN-7684: [~sunilg] Thanks for your update! You are correct, when we view total resource per label, the cluster resource is not correct. And please ignore my comment about the issue happened on version > 2.7.3. This is my mistake to understand. Sorry for the wrong comments! I noticed the [YARN-4484|https://issues.apache.org/jira/browse/YARN-4484] fix target release is 2.8.0, we did not try the 2.8.0 release, we just found the problem on 2.7.1 and 2.7.3. So next we will try your fix on 2.8.0. Any results I will update here. Thanks! > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Fix For: 2.7.3 > > Attachments: YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306156#comment-16306156 ] Sunil G commented on YARN-7684: --- Yes. Using cluster resource might be solving this temporary. However we have partitions and hence cluster resource is not correct when we view total resource per label. {{ this.totalMB = availableMB + allocatedMB;}} For root queue, ideally we should be able to sum up to parent root level. For label, I think we have not correctly calculating queue metrics. YARN-4484 has handled this case, however if its not working now, we need to see how it got broken. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Labels: patch > Fix For: 2.7.3 > > Attachments: YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306077#comment-16306077 ] genericqa commented on YARN-7684: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m 50s{color} | {color:red} Docker failed to build yetus/hadoop:c420dfe. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7684 | | GITHUB PR | https://github.com/apache/hadoop/pull/320 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19054/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Labels: patch > Fix For: 2.7.3 > > Attachments: YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305558#comment-16305558 ] Zhao Yi Ming commented on YARN-7684: [~sunilg] I create a pr for the fix. Could you help review the code? Thanks! Also I saw this problem also happened on version > 2.7.3. Will I need to porting this fix into other upstreams? Thanks! > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Attachments: YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305554#comment-16305554 ] ASF GitHub Bot commented on YARN-7684: -- GitHub user zhaoyim opened a pull request: https://github.com/apache/hadoop/pull/320 YARN-7684. Fix The Total Memory and VCores display in the yarn UI is … The Total Memory and VCores display in the yarn UI is not correct with labeled node Use the cluster resource memory and Vcores info instead of the root queue metrics, availableMB + allocatedMB. and availableVirtualCores + allocatedVirtualCores. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhaoyim/hadoop branch-2.7.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/320.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #320 commit fcec8fd9bde074dda399d56dd426d4fd9aa9c55f Author: zhaoyimDate: 2017-12-28T15:53:22Z YARN-7684. Fix The Total Memory and VCores display in the yarn UI is not correct with labeled node > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Attachments: YARN-7684-branch-2.7.3.001.patch, yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305156#comment-16305156 ] Zhao Yi Ming commented on YARN-7684: @Sunil G Thanks! This is my first try to contribute to hadoop community, once I submit the patch, I will add you as the reviewer. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming > Attachments: yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304418#comment-16304418 ] Sunil G commented on YARN-7684: --- I added to you as a contributor. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming > Attachments: yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7684) The Total Memory and VCores display in the yarn UI is not correct with labeled node
[ https://issues.apache.org/jira/browse/YARN-7684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304404#comment-16304404 ] Zhao Yi Ming commented on YARN-7684: I already fix this issue, I need to assign this issue to myself. > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > > > Key: YARN-7684 > URL: https://issues.apache.org/jira/browse/YARN-7684 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.3 >Reporter: Zhao Yi Ming > Attachments: yarn_issue.pdf > > > The Total Memory and VCores display in the yarn UI is not correct with > labeled node > recreate steps: > 1. should have a hadoop cluster > 2. enabled the yarn Node Labels feature > 3. create a label eg: yarn rmadmin -addToClusterNodeLabels "test" > 4. add a node into the label eg: yarn rmadmin -replaceLabelsOnNode > "zhaoyim02.com=test" > 5. then go to the yarn UI http://:8088/cluster/nodes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org