Hi Omkar, >(which has 40 containers slots.) >> for total cluster? Yes, it was just an hypotetical value though. Below are my real configurations.
>1) yarn-site.xml -> what is the resource memory configured for per node? 12288mb >2) yarn-site.xml -> what is the minimum resource allocation for the cluster? 1024mb min 12288mb max I also have those memory configurations in mapred-site.xml : <property> <name>mapreduce.map.memory.mb</name> <value>5000</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx4g -Djava.awt.headless=true</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>5000</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx4g -Djava.awt.headless=true</value> </property> >3) yarn-resource-manager-log (while starting resource manager "export YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs.. The resulting log is really verbose. Are you searching for something in particular? >4) On RM UI how much total cluster memory is reported (how many total nodes). ( RM UI click on Cluster) So I have 58 active nodes and total memory reported is 696GB which is 58x12 as expected. I have 93 containers running instead of 116 I would expect (my job has 2046 maps so it could use all 116 containers). Here is a copy past of what I have in the scheduler tab: *Queue State: * RUNNING *Used Capacity: * 99.4% *Absolute Capacity: * 100.0% *Absolute Max Capacity: * 100.0% *Used Resources: * *Num Active Applications: * 1 *Num Pending Applications: * 0 *Num Containers: * 139 *Max Applications: * 10000 *Max Applications Per User: * 10000 *Max Active Applications: * 70 *Max Active Applications Per User: * 70 *Configured Capacity: * 100.0% *Configured Max Capacity: * 100.0% *Configured Minimum User Limit Percent: * 100% *Configured User Limit Factor: * 1.0 *Active users: * xxx <Memory: 708608 (100.00%), vCores: 139 (100.00%), Active Apps: 1, Pending Apps: 0> I don't know where the 139 containers value is comming from. >5) which scheduler you are using? Capacity/Fair/FIFO I did not set yarn.resourcemanager.scheduler.class so apparently the default is Capacity. >6) have you configured any user limits/ queue capacity? (please add details). No. >7) All requests you are making at same priority or with different priorities? (Ideally it will not matter but want to know). I don't set any priority. Thanks for your help. Antoine Vandecreme On Friday, September 20, 2013 12:20:38 PM Omkar Joshi wrote: > Hi, > > few more questions > > (which has 40 containers slots.) >> for total cluster? Please give below > details > > for cluster > 1) yarn-site.xml -> what is the resource memory configured for per node? > 2) yarn-site.xml -> what is the minimum resource allocation for the cluster? > 3) yarn-resource-manager-log (while starting resource manager "export > YARN_ROOT_LOGGER=DEBUG,RFA").. I am looking for debug logs.. > 4) On RM UI how much total cluster memory is reported (how many total > nodes). ( RM UI click on Cluster) > 5) which scheduler you are using? Capacity/Fair/FIFO > 6) have you configured any user limits/ queue capacity? (please add > details). > 7) All requests you are making at same priority or with different > priorities? (Ideally it will not matter but want to know). > > Please let us know all the above details. Thanks. > > > Thanks, > Omkar Joshi > *Hortonworks Inc.* <http://www.hortonworks.com> > > > On Fri, Sep 20, 2013 at 6:55 AM, Antoine Vandecreme < > > antoine.vandecr...@nist.gov> wrote: > > Hello Omkar, > > > > Thanks for your reply. > > > > Yes, all 4 points are corrects.