[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-24 Thread Haibo Chen (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662393#comment-16662393
 ] 

Haibo Chen commented on YARN-8911:
--

Thanks [~elgoiri] for the review! I have checked 02 patch into trunk.

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch, 
> YARN-8911.02.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-24 Thread Hudson (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662395#comment-16662395
 ] 

Hudson commented on YARN-8911:
--

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15306 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/15306/])
YARN-8911. ContainerScheduler incorrectly uses percentage number as the 
(haibochen: rev 766b78ee074df29d5acd2ba2face5712dd20437f)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/AllocationBasedResourceUtilizationTracker.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerRecovery.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestAllocationBasedResourceUtilizationTracker.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceUtilization.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java


> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch, 
> YARN-8911.02.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661585#comment-16661585
 ] 

Íñigo Goiri commented on YARN-8911:
---

Thanks [~haibochen] for the updated version.
[^YARN-8911.02.patch] LGTM.

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch, 
> YARN-8911.02.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661481#comment-16661481
 ] 

Hadoop QA commented on YARN-8911:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 23s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 23s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
43s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
41s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8911 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12945282/YARN-8911.02.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux a59f6e57c342 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 635786a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/22309/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
|

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Haibo Chen (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661341#comment-16661341
 ] 

Haibo Chen commented on YARN-8911:
--

Good point. I have modified TestContainerSchedulerRecovery to do that.

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661190#comment-16661190
 ] 

Hadoop QA commented on YARN-8911:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 26s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 43s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 12s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 47s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8911 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12945111/YARN-8911.01.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 73c7fb82ce15 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 93fb3b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661096#comment-16661096
 ] 

Íñigo Goiri commented on YARN-8911:
---

Shouldn't the container scheduler test then be affected by this change?
If not, we should add some test to make sure the value is something >1.

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-23 Thread Haibo Chen (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660988#comment-16660988
 ] 

Haibo Chen commented on YARN-8911:
--

{quote} Is there any test that would track this from the container to the RM?
{quote}
I have not been able to find any.

With that said, the change included in this patch has nothing to do with the 
code path that collects utilization of containers and sends to RM.  Hence, such 
tests won't cover the changes here.

The containers and node utilization are still collected by containers monitor 
and NodeResourceMonitor ,and sent to RM in the heartbeat. What's changed in 
containers monitor is a helper methods that gets called by Container Scheduler 
only, which tracks containers utilization internally. There are some test 
coverage on the container utilization maintained by Container Scheduler already.

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread JIRA 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659851#comment-16659851
 ] 

Íñigo Goiri commented on YARN-8911:
---

I don't think we have very good coverage of this code.
Is there any test that would track this from the container to the RM?

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8911) ContainerScheduler incorrectly uses percentage number as the cpu resource utlization

2018-10-22 Thread Haibo Chen (JIRA) 


[ 
https://issues.apache.org/jira/browse/YARN-8911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659739#comment-16659739
 ] 

Haibo Chen commented on YARN-8911:
--

Uploaded a new patch that

1) updates the ResourceUtilization cpu javadoc to reduce ambiguity.

2) updates use of ResourceUtilization in YARN-4597

3 ) reverts the core YARN-6870 changes which fixed a floating point inaccuracy 
issue that no longer applies.

[~elgoiri] [~asuresh] Would appreciate your reviews. CC [~brookz].

> ContainerScheduler incorrectly uses percentage number as the cpu resource 
> utlization
> 
>
> Key: YARN-8911
> URL: https://issues.apache.org/jira/browse/YARN-8911
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8911.00.patch, YARN-8911.01.patch
>
>
> *UPDATE*:  *per discussion below, the cpu resource utlization 
> (ResourceUtilzation.cpu) is incorrectly documented as a percentage number in 
> [0, 1.0] in the java doc. It is however, used as the number of vcores used 
> throughout the system.  YARN-4597 is a victim of the  misleading java doc.*
>  
>  
> ResourceUtilization represents the cpu utilization with a float number in [0, 
> 1.0], i.e. the percentage of cpu usage across the node.  However, when 
> Containers Monitor tracks the total aggregate resource utilization of all 
> containers, it adds up the total number of vcores used by all running 
> containers.
>  
> (If you have 6 containers running, each consuming 1 vcore, you'd see the 
> aggregated cpu container utilization being 6.0, but it's supposed to be 
> always between 0 and 1.0)   See [the 
> code|https://github.com/apache/hadoop/blob/beb850d8f7f1fefa7a6d9502df2b4a4eea372523/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L672]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org