Hi Flocks,
Since, last two days I am about to configure Capacity Scheduler. Here, how I have struggling L…. I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4 Queue’s as per the Capacity Scheduler Documentation and those queues shown in RM UI. After configuration I tried to run Jobs, Cap Scheduler not identified that queue’s. where I have check queues list with “mapred queue –list”, which showing all configured Q’s. I wrote a mail’s to groups for solution, Mr.Olivier has been given some idea about that, based on his views I dig more. After I went to all the RM log, Cap Scheduler initiating only default “default”, I have tested with *default queue* it works for me. And I have created one more queue called “dev”, in this Queue User unable to run the jobs and its unable to identifying users Queue. I have attached Cap Scheduler configuration file for your information. Some O/P for ur information. *[user@host ~]$ mapred queue -list* *13/11/27 09:26:38 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.* *13/11/27 09:26:38 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.* *======================* *Queue Name : dev* *Queue State : running* *Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5, CurrentCapacity: 0.0* *======================* *Queue Name : default* *Queue State : running* *Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0* *RM log Scheduler loading info:* 2013-11-27 08:54:58,521 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0, asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS: ADMINISTER_QUEUE: 2013-11-27 08:54:58,521 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Initialized parent-queue root name=root, fullname=root 2013-11-27 08:54:58,534 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing default* capacity = 0.7 [= (float) configuredCapacity / 100 ] asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ] maxCapacity = 1.0 [= configuredMaxCapacity ] absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] userLimit = 100 [= configuredUserLimit ] userLimitFactor = 1.0 [= configuredUserLimitFactor ] maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ] maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ] usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)] absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory] maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ] maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ] numContainers = 0 [= currentNumContainers ] state = RUNNING [= configuredState ] acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE: [= configuredAcls ] 2013-11-27 08:54:58,534 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized queue: default: capacity=0.7, absoluteCapacity=0.7, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=0, numContainers=0 2013-11-27 08:54:58,543 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: *Initializing dev* capacity = 0.3 [= (float) configuredCapacity / 100 ] asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ] maxCapacity = 0.5 [= configuredMaxCapacity ] absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined, (parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ] userLimit = 100 [= configuredUserLimit ] userLimitFactor = 1.0 [= configuredUserLimitFactor ] maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or (int)(configuredMaximumSystemApplications * absoluteCapacity)] maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit / 100.0f) * userLimitFactor) ] maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1) ] maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory / minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ] maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications * (userLimit / 100.0f) * userLimitFactor),1) ] usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory * absoluteCapacity)] absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory] maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ] minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory - minimumAllocationMemory) / maximumAllocationMemory ] numContainers = 0 [= currentNumContainers ] state = RUNNING [= configuredState ] acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE: [= configuredAcls ] Can you guys please confirm, did I miss anything on configurations part or is there any bug persist on 2.0.0? Thanks Munna
<configuration> <property> <name>yarn.scheduler.capacity.maximum-applications</name> <value>10000</value> <description> Maximum number of applications that can be pending and running. </description> </property> <property> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> <value>0.1</value> <description> Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications. </description> </property> <property> <name>yarn.scheduler.capacity.resource-calculator</name> <value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value> <description> The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. </description> </property> <property> <name>yarn.scheduler.capacity.root.queues</name> <value>default,dev</value> </property> <property> <name>yarn.scheduler.capacity.root.default.capacity</name> <value>70</value> </property> <property> <name>yarn.scheduler.capacity.root.dev.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.default.user-limit-factor</name> <value>1</value> <description> Default queue user limit a percentage from 0.0 to 1.0. </description> </property> <property> <name>yarn.scheduler.capacity.root.dev.user-limit-factor</name> <value>1</value> <description> Default queue user limit a percentage from 0.0 to 1.0. </description> </property> <property> <name>yarn.scheduler.capacity.root.default.maximum-capacity</name> <value>100</value> <description> The maximum capacity of the default queue. </description> </property> <property> <name>yarn.scheduler.capacity.root.dev.maximum-capacity</name> <value>50</value> <description> The maximum capacity of the default queue. </description> </property> <property> <name>yarn.scheduler.capacity.root.default.state</name> <value>RUNNING</value> <description> The state of the default queue. State can be one of RUNNING or STOPPED. </description> </property> <property> <name>yarn.scheduler.capacity.root.dev.state</name> <value>RUNNING</value> <description> The state of the default queue. State can be one of RUNNING or STOPPED. </description> </property> <property> <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name> <value>yarn,mapred</value> </property> <property> <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name> <value>smunnavar,test</value> </property> <property> <name>yarn.scheduler.capacity.node-locality-delay</name> <value>-1</value> <description> Number of missed scheduling opportunities after which the CapacityScheduler attempts to schedule rack-local containers. Typically this should be set to number of racks in the cluster, this feature is disabled by default, set to -1. </description> </property> <property> <name>yarn.scheduler.capacity.root.acl_submit_applications</name> <value> </value> </property> <property> <name>yarn.scheduler.capacity.root.acl_administer_queue</name> <value> </value> </property> </configuration>