[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599379#comment-17599379 ] zhengchenyu commented on YARN-5936: --- For work change, I miss long long time. In fact, we need make sure all task of the container in same task group. Use task group as linux shed entity, will assure fairness. (Note: here task is linux task). I redo the experiment in hadoop-3.2.1, linux kernel is 3.10.0-862.el7.x86_64. The containers nearly fairness. Note: original experiment is not serious, redo it! {code:java} ## The first container is 100 thread, the second is 150 thread. The logical core of host is 64. hadoop jar /home/ke/bin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.1.jar org.apache.hadoop.yarn.applications.distributedshell.Client -jar /home/ke/bin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.1.jar -shell_script ./run.sh -shell_args 100 -num_containers 1 -container_memory 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 hadoop jar /home/ke/bin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.1.jar org.apache.hadoop.yarn.applications.distributedshell.Client -jar /home/ke/bin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.1.jar -shell_script ./run.sh -shell_args 150 -num_containers 1 -container_memory 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10{code} The cpu usage of containers nearly fairness. So close it. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override >
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248592#comment-16248592 ] zhengchenyu commented on YARN-5936: --- We open cpu strict mode, because the container can't use the free resource. But we close cpu strict mode, it's not fair for every container. If i set the cpu bandwidth of cgroup dynamically, I think this problem is solved. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250021#comment-16250021 ] Miklos Szegedi commented on YARN-5936: -- Another option for the future is the use of the cgroup pids subsystem on newer kernels. The main reason fairness is not enforced in non-strict mode, is that it allows the container to run as many threads with the same cgroup and weight as needed. You can limit the amount of threads with the pids namespace, so that the effective overall container weight becomes weight*pids_limit. The drawback of this approach is that it limits multitasking and the number of launcher processes. The possible ideal value of pids_limit is the /, so that we do not starve single threaded containers. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706129#comment-15706129 ] Miklos Szegedi commented on YARN-5936: -- Thank you for reporting this issue [~zhengchenyu]! When you mentioned "But the cpu bandwidth of cgroup would lead to bad performance in our experience.", do you mean that it is due to the design that it limits the CPU usage to the vCore share affecting overall utilization, or do you mean that the container got less resources than what was assigned to it? In other words, is this a remark of the strict CPU cgroup design or the implementation? Thank you! > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708991#comment-15708991 ] Weiwei Yang commented on YARN-5936: --- Hello [~zhengchenyu] How many cores your system has, can you attach the output of top command that contains each core usages (by type 1 after top)? It looks like it was not fair because your system still has plenty of resources, so each container gets whatever it needs to run happily. It only maintains fairness relatively. According to the [cgroups document|https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html] bq. cpu.shares contains an integer value that specifies a relative share of CPU time available to the tasks in a cgroup ... Note that shares of CPU time are distributed per all CPU cores on multi-core systems. Even if a cgroup is limited to less than 100% of CPU on a multi-core system, it may use 100% of each individual CPU core. What you are asking for (container level strict limit) looks like YARN-810, unfortunately it has been left out there for quite a long time already. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlas
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721247#comment-15721247 ] zhengchenyu commented on YARN-5936: --- I think two reason both affect the performance, but I can't evaluate which is the major reason. First,The linux kernel source code of cpu bandwidth will add too many timer, and add more function to be called. Secondly, limit utilization ratio will lead to bad performance. Closing the cpu bandwith limit is inevitabe. Here I only wanna to a idea that keep justice when only use cpu share. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721341#comment-15721341 ] zhengchenyu commented on YARN-5936: --- You didn't catch my meaning! In fact, I knew the reason of the unfairness. The processes and threads has the same level of scheduling. In Linux-3.10, task are scheduled by a red–black tree, and update the red–black tree periodically or manually(by other calling). the left-most is next task to be scheduled. The red-black tree is update by this formula: curr->vruntime+=delta_exec*nice_0_load/curr->load.weight here curr->load.weight is just cpu.share. So Mulit Thread obtain more cpu than Single-Thread because of more child thread are participating the scheduler. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755361#comment-15755361 ] Miklos Szegedi commented on YARN-5936: -- Thank you, for the reply [~zhengchenyu]! {quote} First,The linux kernel source code of cpu bandwidth will add too many timer, and add more function to be called. Secondly, limit utilization ratio will lead to bad performance. {quote} I did an experiment with he following cpu heavy app: {code} //a.c int main() { int i; int j = 0; for (i = 0; i < 10; ++i) { j++; } return j & 1; } {code} I ran it in parallel in a single cgroup, multiple cgroups and multipre cgroups with CPU throttling enabled on a single CPU. {code} for j in `seq 1 10`; do export i=$j;sh -c 'time ./a.out&'; done for j in `seq 1 10`; do export i=$j;sh -c 'echo $$ >/cgroup/cpu/$i/tasks;echo -1 >/cgroup/cpu/$i/cpu.cfs_quota_us;time ./a.out&'; done for j in `seq 1 10`; do export i=$j;sh -c 'echo $$ >/cgroup/cpu/$i/tasks;echo 1 >/cgroup/cpu/$i/cpu.cfs_quota_us;time ./a.out&'; done {code} The runtime in the first case (no cgroups) was 24.7154, the second (no group throttle) was 24.6907 seconds on average, the runtime in the latter case was 24.7469 respectively. The difference less than 0.25% in these cases. I ran it a few more times and I received very similar numbers. This means to me that what you are seeing is the utilization drop, if the container group limits the CPU usage and not an inefficiency in the Linux kernel. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override >
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760010#comment-15760010 ] zhengchenyu commented on YARN-5936: --- Because your program is a single-thread. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761877#comment-15761877 ] Miklos Szegedi commented on YARN-5936: -- I run 10 of them in parallel. {quote} First,The linux kernel source code of cpu bandwidth will add too many timer, and add more function to be called. {quote} Are you saying you see this only if an app is running multiple threads and not multiple processes? > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762998#comment-15762998 ] Miklos Szegedi commented on YARN-5936: -- Indeed, I see some 6% performance loss in the second case above compared to the first. This is when I move away from the root cgroup and use the hierarchy but do not use the limit, yet. This happens, when I am running 10 processes in different cgroups with 100 threads each. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763020#comment-15763020 ] zhengchenyu commented on YARN-5936: --- But your "time" command is only related to it's own program, every program is single thread. My question is below: two process that has different numbers of threads has different ability of schedule, though they have the same cpu.share. I know the reason, but I don't have a proper suggestion to avoid this problem. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763021#comment-15763021 ] zhengchenyu commented on YARN-5936: --- But your "time" command is only related to it's own program, every program is single thread. My question is below: two process that has different numbers of threads has different ability of schedule, though they have the same cpu.share. I know the reason, but I don't have a proper suggestion to avoid this problem. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763023#comment-15763023 ] zhengchenyu commented on YARN-5936: --- But your "time" command is only related to it's own program, every program is single thread. My question is below: two process that has different numbers of threads has different ability of schedule, though they have the same cpu.share. I know the reason, but I don't have a proper suggestion to avoid this problem. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763042#comment-15763042 ] zhengchenyu commented on YARN-5936: --- I don't care the performance loss of cpu bandwith, because I know it must happen. I only wanna know the method of keep fair without cpu bandwith. > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Fix For: 2.7.1 > > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { > @Override > public void run() { > // TODO Auto-generated method stub > int j=0; > while(true){j++;} > } > }).start(); > } > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[ https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806611#comment-15806611 ] Miklos Szegedi commented on YARN-5936: -- In the latest test I used 100 threads per program, I just did not share the code. They run in parallel, so the sum of time command results measures, whether the whole set spent time in additional CPU cycles other than the activity loop. The reason I checked, is to ask whether you like a solution that uses {{cpu.cfs_quota_us}}. I could imagine a dynamic cfs algorithm like the following. A timer callback with a certain period could do: {code} if CPU is saturated for each container if previous usage > fair share limit to fair share else release all limits {code} It has drawbacks. It only works with saturated CPU, when not much time is spent waiting on I/O. It has a delay, since it works on historic data. This means also that it adds some utilization loss, which can be larger with multiple cores. On the other hand, it provides the requested fairness, when the CPU is saturated. Does your node have multiple cores? The algorithm may not help much in that case. For example there are 8 cores. One container runs 8 threads, one container runs 2 threads. The fair share requested is 50%-50%. Without throttling the two containers will share 80%-20%. Even, if we set the fair share by throttling, when the cores are saturated, the usage will be 50%/25% when the quota is applied, so there is a utilization loss for a period. Now then, the algorithm may get more complicated... > when cpu strict mode is closed, yarn couldn't assure scheduling fairness > between containers > --- > > Key: YARN-5936 > URL: https://issues.apache.org/jira/browse/YARN-5936 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.1 > Environment: CentOS7.1 >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 1m > Remaining Estimate: 1m > > When using LinuxContainer, the setting that > "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is > true could assure scheduling fairness with the cpu bandwith of cgroup. But > the cpu bandwidth of cgroup would lead to bad performance in our experience. > Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to > assure scheduling fairness, but it is not completely effective. For example, > There are two container that have same vcore(means same cpu.share), one > container is single-threaded, the other container is multi-thread. the > multi-thread will have more CPU time, It's unreasonable! > Here is my test case, I submit two distributedshell application. And two > commmand are below: > {code} > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > hadoop jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > org.apache.hadoop.yarn.applications.distributedshell.Client -jar > share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar > -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory > 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10 > {code} > here show the cpu time of the two container: > {code} > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND > 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java > 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java > 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java >77 root rt 0 0 0 0 S 0.3 0.0 0:00.74 > migration/1 > {code} > We find the cpu time of Muliti-Thread are ten times than the cpu time of > Single-Thread, though the two container have same cpu.share. > notes: > run.sh > {code} > java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1 > {code} > loop.java > {code} > package loop; > public class loop { > public static void main(String[] args) { > // TODO Auto-generated method stub > int loop = 1; > if(args.length>=1) { > System.out.println(args[0]); > loop = Integer.parseInt(args[0]); > } > for(int i=0;i System.out.println("start thread " + i); > new Thread(new Runnable() { >