[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662393#comment-16662393 ] Haibo Chen commented on YARN-8911: -- Thanks [~elgoiri] for the review! I have checked 02 patch into trunk. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch, > YARN-8911.02.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662395#comment-16662395 ] Hudson commented on YARN-8911: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15306 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15306/]) YARN-8911. ContainerScheduler incorrectly uses percentage number as the (haibochen: rev 766b78ee074df29d5acd2ba2face5712dd20437f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerRecovery.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestAllocationBasedResourceUtilizationTracker.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceUtilization.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch, > YARN-8911.02.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661585#comment-16661585 ] Íñigo Goiri commented on YARN-8911: --- Thanks [~haibochen] for the updated version. [^YARN-8911.02.patch] LGTM. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch, > YARN-8911.02.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661481#comment-16661481 ] Hadoop QA commented on YARN-8911: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 23s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 23s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 43s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 99m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 | | JIRA Issue | YARN-8911 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12945282/YARN-8911.02.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a59f6e57c342 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 635786a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22309/testReport/ | | Max. process+thread count | 306 (vs. ulimit of
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661341#comment-16661341 ] Haibo Chen commented on YARN-8911: -- Good point. I have modified TestContainerSchedulerRecovery to do that. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661190#comment-16661190 ] Hadoop QA commented on YARN-8911: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 24s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 43s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 12s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 | | JIRA Issue | YARN-8911 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12945111/YARN-8911.01.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 73c7fb82ce15 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 93fb3b4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | unit | https://bu
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661096#comment-16661096 ] Íñigo Goiri commented on YARN-8911: --- Shouldn't the container scheduler test then be affected by this change? If not, we should add some test to make sure the value is something >1. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660988#comment-16660988 ] Haibo Chen commented on YARN-8911: -- {quote} Is there any test that would track this from the container to the RM? {quote} I have not been able to find any. With that said, the change included in this patch has nothing to do with the code path that collects utilization of containers and sends to RM. Hence, such tests won't cover the changes here. The containers and node utilization are still collected by containers monitor and NodeResourceMonitor ,and sent to RM in the heartbeat. What's changed in containers monitor is a helper methods that gets called by Container Scheduler only, which tracks containers utilization internally. There are some test coverage on the container utilization maintained by Container Scheduler already. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659851#comment-16659851 ] Íñigo Goiri commented on YARN-8911: --- I don't think we have very good coverage of this code. Is there any test that would track this from the container to the RM? > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization
[ https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659739#comment-16659739 ] Haibo Chen commented on YARN-8911: -- Uploaded a new patch that 1) updates the ResourceUtilization cpu javadoc to reduce ambiguity. 2) updates use of ResourceUtilization in YARN-4597 3 ) reverts the core YARN-6870 changes which fixed a floating point inaccuracy issue that no longer applies. [~elgoiri] [~asuresh] Would appreciate your reviews. CC [~brookz]. > ContainerScheduler incorrectly uses percentage number as the cpu resource > utlization > > > Key: YARN-8911 > URL: https://issues.apache.org/jira/browse/YARN-8911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8911.00.patch, YARN-8911.01.patch > > > *UPDATE*: *per discussion below, the cpu resource utlization > (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in > [0, 1.0] in the java doc. It is however, used as the number of vcores used > throughout the system. YARN-4597 is a victim of the misleading java doc.* > > > ResourceUtilization represents the cpu utilization with a float number in [0, > 1.0], i.e. the percentage of cpu usage across the node. However, when > Containers Monitor tracks the total aggregate resource utilization of all > containers, it adds up the total number of vcores used by all running > containers. > > (If you have 6 containers running, each consuming 1 vcore, you'd see the > aggregated cpu container utilization being 6.0, but it's supposed to be > always between 0 and 1.0) See [the > code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org