Hi Flocks,


Since, last two days I am about to configure Capacity Scheduler. Here, how
I have struggling L….



I am using Hadoop 2.0.0 and YARN 2.0.0(CDH4). Initially I have created 4
Queue’s as per the Capacity Scheduler Documentation and those queues shown
in RM UI.



After configuration I tried to run Jobs, Cap Scheduler not identified that
queue’s. where I have check queues list with “mapred queue –list”, which
showing all configured Q’s.



I wrote a mail’s to groups for solution, Mr.Olivier has been given some
idea about that, based on his views I dig more.



After I went to all the RM log, Cap Scheduler initiating only default
“default”, I have tested with *default queue* it works for me. And I have
created one more queue called “dev”, in this Queue User unable to run the
jobs and its unable to identifying users Queue.



I have attached Cap Scheduler configuration file for your information. Some
O/P for ur information.



*[user@host ~]$ mapred queue -list*

*13/11/27 09:26:38 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.*

*13/11/27 09:26:38 INFO service.AbstractService:
Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.*

*======================*

*Queue Name : dev*

*Queue State : running*

*Scheduling Info : Capacity: 30.000002, MaximumCapacity: 0.5,
CurrentCapacity: 0.0*

*======================*

*Queue Name : default*

*Queue State : running*

*Scheduling Info : Capacity: 70.0, MaximumCapacity: 1.0, CurrentCapacity:
0.0*



*RM log Scheduler loading info:*

2013-11-27 08:54:58,521 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
root, capacity=1.0, asboluteCapacity=1.0, maxCapacity=1.0,
asboluteMaxCapacity=1.0, state=RUNNING, acls=SUBMIT_APPLICATIONS:
ADMINISTER_QUEUE:

2013-11-27 08:54:58,521 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Initialized parent-queue root name=root, fullname=root

2013-11-27 08:54:58,534 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
*Initializing
default*

capacity = 0.7 [= (float) configuredCapacity / 100 ]

asboluteCapacity = 0.7 [= parentAbsoluteCapacity * capacity ]

maxCapacity = 1.0 [= configuredMaxCapacity ]

absoluteMaxCapacity = 1.0 [= 1.0 maximumCapacity undefined,
(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]

userLimit = 100 [= configuredUserLimit ]

userLimitFactor = 1.0 [= configuredUserLimitFactor ]

maxApplications = 7000 [= configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]

maxApplicationsPerUser = 7000 [= (int)(maxApplications * (userLimit /
100.0f) * userLimitFactor) ]

maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]

maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]

maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
(userLimit / 100.0f) * userLimitFactor),1) ]

usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
absoluteCapacity)]

absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]

maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]

minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]

maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]

minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]

numContainers = 0 [= currentNumContainers ]

state = RUNNING [= configuredState ]

acls = SUBMIT_APPLICATIONS:mapred,yarn ADMINISTER_QUEUE:  [= configuredAcls
]



2013-11-27 08:54:58,534 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Initialized queue: default: capacity=0.7, absoluteCapacity=0.7,
usedResources=<memory:0, vCores:0>usedCapacity=0.0,
absoluteUsedCapacity=0.0, numApps=0, numContainers=0

2013-11-27 08:54:58,543 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
*Initializing
dev*

capacity = 0.3 [= (float) configuredCapacity / 100 ]

asboluteCapacity = 0.3 [= parentAbsoluteCapacity * capacity ]

maxCapacity = 0.5 [= configuredMaxCapacity ]

absoluteMaxCapacity = 0.5 [= 1.0 maximumCapacity undefined,
(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise ]

userLimit = 100 [= configuredUserLimit ]

userLimitFactor = 1.0 [= configuredUserLimitFactor ]

maxApplications = 3000 [= configuredMaximumSystemApplicationsPerQueue or
(int)(configuredMaximumSystemApplications * absoluteCapacity)]

maxApplicationsPerUser = 3000 [= (int)(maxApplications * (userLimit /
100.0f) * userLimitFactor) ]

maxActiveApplications = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) * maxAMResourcePerQueuePercent * absoluteMaxCapacity),1)
]

maxActiveAppsUsingAbsCap = 1 [= max((int)ceil((clusterResourceMemory /
minimumAllocation) *maxAMResourcePercent * absoluteCapacity),1) ]

maxActiveApplicationsPerUser = 1 [= max((int)(maxActiveApplications *
(userLimit / 100.0f) * userLimitFactor),1) ]

usedCapacity = 0.0 [= usedResourcesMemory / (clusterResourceMemory *
absoluteCapacity)]

absoluteUsedCapacity = 0.0 [= usedResourcesMemory / clusterResourceMemory]

maxAMResourcePerQueuePercent = 0.1 [= configuredMaximumAMResourcePercent ]

minimumAllocationFactor = 0.875 [= (float)(maximumAllocationMemory -
minimumAllocationMemory) / maximumAllocationMemory ]

numContainers = 0 [= currentNumContainers ]

state = RUNNING [= configuredState ]

acls = SUBMIT_APPLICATIONS:user,test ADMINISTER_QUEUE:  [= configuredAcls ]



Can you guys please confirm, did I miss anything on configurations part or
is there any bug persist on 2.0.0?



Thanks

Munna
<configuration>

<property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare
      multi-dimensional resources such as Memory, CPU etc.
    </description>
</property>
<property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,dev</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>70</value>
</property>
<property>
    <name>yarn.scheduler.capacity.root.dev.capacity</name>
    <value>30</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.dev.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.dev.maximum-capacity</name>
    <value>50</value>
    <description>
      The maximum capacity of the default queue. 
    </description>
</property>
  
<property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
</property>
<property>
    <name>yarn.scheduler.capacity.root.dev.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
  <value>yarn,mapred</value>
</property>

<property>
        <name>yarn.scheduler.capacity.root.dev.acl_submit_applications</name>
		<value>smunnavar,test</value>
</property>


<property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>-1</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler 
      attempts to schedule rack-local containers. 
      Typically this should be set to number of racks in the cluster, this 
      feature is disabled by default, set to -1.
    </description>
</property>

<property>
      <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
      <value> </value>
</property>

<property>
      <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
      <value> </value>
</property>

</configuration>

Reply via email to