I believe it's behaving as expected. It will spawn 64 containers because that's how much memory you have available. The vcores isn't harshly enforced since CPUs can be elastic. This blog from cloudera explain how to enforce CPU limits using CGroups. http://blog.cloudera.com/blog/2013/12/managing-multiple-resources-in-hadoop-2-with-yarn/
On Tue, May 27, 2014 at 8:56 PM, hari <harib...@gmail.com> wrote: > The issue was not related the configuration related to containers. Due to > misconfiguration, the Application master was not able to contact > resourcemanager > causing in the 1 container problem. > > However, the total containers allocated still is not as expected. The > configuration settings > should have resulted in 16 containers per node, but it is allocating 64 > containers per node. > > Reiterating the config parameters here again: > > mapred-site.xml > mapreduce.map.cpu.vcores = 1 > mapreduce.reduce.cpu.vcores = 1 > mapreduce.map.memory.mb = 1024 > mapreduce.reduce.memory.mb = 1024 > mapreduce.map.java.opts = -Xmx1024m > mapreduce.reduce.java.opts = -Xmx1024m > > yarn.xml > yarn.nodemanager.resource.memory-mb = 65536 > yarn.nodemanager.resource.cpu-vcores = 16 > yarn.scheduler.minimum-allocation-mb = 1024 > yarn.scheduler.maximum-allocation-mb = 2048 > yarn.scheduler.minimum-allocation-vcores = 1 > yarn.scheduler.maximum-allocation-vcores = 1 > > Is there anything else that might be causing this problem ? > > thanks, > hari > > > > > > On Tue, May 27, 2014 at 3:31 AM, hari <harib...@gmail.com> wrote: > >> Hi, >> >> When using YARN 2.2.0 version, only 1 container is created >> for an application in the entire cluster. >> The single container is created at an arbitrary node >> for every run. This happens when running any application from >> the examples jar (e.g., wordcount). Currently only one application is >> run at a time. The input datasize is > 200GB. >> >> I am setting custom values that affect concurrent container count. >> These config parameters were mostly taken from: >> >> http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ >> These wasn't much description elsewhere on how the container count would >> be >> decided. >> >> The settings are: >> >> mapred-site.xml >> mapreduce.map.cpu.vcores = 1 >> mapreduce.reduce.cpu.vcores = 1 >> mapreduce.map.memory.mb = 1024 >> mapreduce.reduce.memory.mb = 1024 >> mapreduce.map.java.opts = -Xmx1024m >> mapreduce.reduce.java.opts = -Xmx1024m >> >> yarn.xml >> yarn.nodemanager.resource.memory-mb = 65536 >> yarn.nodemanager.resource.cpu-vcores = 16 >> >> From these settings, each node should be running 16 containers. >> >> Let me know if there might be something else affecting the container >> count. >> >> thanks, >> hari >> >> >> >> >> >> >
