[
https://issues.apache.org/jira/browse/YARN-600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681486#comment-13681486
]
Chris Riccomini commented on YARN-600:
--------------------------------------
Hey Sandy,
I just applied your patch to my local YARN, and can verify that it appears to
be working.
{noformat}
$ cat container_1371061837111_0001_01_000002/cpu.shares
1024
$ cat container_1371061837111_0002_01_000002/cpu.shares
32768
{noformat}
I have 8 of the second type of container (32768 cpu shares) on an 8 core
machine. When running 8 * 32768 and 1 * 1024, I get a top that looks like this:
{noformat}
1404 criccomi 20 0 1022m 108m 13m S 98.1 0.2 2:33.03
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0005/container_1371061837111_0005_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3192 criccomi 20 0 1022m 109m 13m S 98.1 0.2 2:25.93
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0009/container_1371061837111_0009_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
428 criccomi 20 0 1022m 109m 13m S 97.7 0.2 2:36.41
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0004/container_1371061837111_0004_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3022 criccomi 20 0 1022m 110m 13m S 97.2 0.2 2:29.74
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0007/container_1371061837111_0007_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
32443 criccomi 20 0 1022m 109m 13m S 95.1 0.2 2:40.17
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0003/container_1371061837111_0003_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
2850 criccomi 20 0 1022m 107m 13m S 93.6 0.2 2:31.09
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0006/container_1371061837111_0006_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
3112 criccomi 20 0 1022m 108m 13m S 93.2 0.2 2:25.54
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0008/container_1371061837111_0008_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
31038 criccomi 20 0 1022m 109m 13m S 84.5 0.2 3:07.39
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0002/container_1371061837111_0002_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
29451 criccomi 20 0 1925m 249m 13m S 16.3 0.4 0:33.29
/export/apps/jdk/JDK-1_6_0_27/bin/java -Dproc_nodemanager -Xmx1000m -server
-Dhadoop.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs
-Dyarn.log.dir=/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs
-Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.lo
30447 criccomi 20 0 1022m 109m 13m S 3.7 0.2 1:28.42
/export/apps/jdk/JDK-1_6_0_27/bin/java -Xmx160M -XX:+PrintGCDateStamps
-Xloggc:/home/criccomi/Downloads/hadoop-2.0.5-alpha/logs/userlogs/application_1371061837111_0001/container_1371061837111_0001_01_000002/gc.log
-Dlog4j.configuration=file:/tmp/hadoop-cri
{noformat}
The column that starts with 98 is the CPU column. As you can see,
container_1371061837111_0001_01_000002 is only taking 3% CPU, while the other
processes are taking 100%. They're all doing the same thing to burn up CPU, so
it appears the CGroups are throttling my 1024 container as expected.
Cheers,
Chris
> Hook up cgroups CPU settings to the number of virtual cores allocated
> ---------------------------------------------------------------------
>
> Key: YARN-600
> URL: https://issues.apache.org/jira/browse/YARN-600
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.3-alpha
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Attachments: YARN-600.patch
>
>
> YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and
> introduced CPU scheduling in the capacity scheduler, and YARN-326 will
> introduce it in the fair scheduler. The number of virtual cores allocated to
> a container should be used to weight the number of cgroups CPU shares given
> to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
