[
https://issues.apache.org/jira/browse/YARN-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442200#comment-17442200
]
Tamas Domok edited comment on YARN-10922 at 11/11/21, 10:04 AM:
----------------------------------------------------------------
*I verified that Queue ACL can't be configured via the leaf-queue-template,
indeed they are inherited from the parent.*
*I verified that Absolute Resource can't be configured via the
leaf-queue-template.*
*Details on Queue ACL:*
1. Do not allow anybody to submit/admin apps on the root queue
{code:java}
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value> </value>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_administer_queue</name>
<value> </value>
</property> {code}
2. Allow only nottdomok, testuser and TestUser for the parent1 queue
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.acl_submit_applications</name>
<value>nottdomok,testuser,TestUser</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.acl_administer_queue</name>
<value>nottdomok,testuser,TestUser</value>
</property> {code}
3. Try to allow tdomok using the leaf-queue-template
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.acl_submit_applications</name>
<value>tdomok</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.acl_administer_queue</name>
<value>tdomok</value>
</property> {code}
4. Results: TestUser can submit a job to the dynamically created TestUser
queue, while the user tdomok can't submit to the tdomok queue which would be
dynamically created.
{code:java}
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException:
org.apache.hadoop.security.AccessControlException: User tdomok does not have
permission to submit application_1636622725698_0001 to queue tdomok {code}
*Details on Absolute Resource*
1. Configuration
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.capacity</name>
<value>[memory=4096,vcores=8]</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.capacity</name>
<value>[memory=1024,vcores=4]</value>
</property> {code}
2. curl "http://localhost:8088/ws/v1/cluster/scheduler"; --verbose | jq &>
cfg.xml
{code:java}
"queues": {
"queue": [
{
"queuePath": "root.parent1",
"capacity": 50,
"usedCapacity": 0,
"maxCapacity": 100,
"absoluteCapacity": 50,
"absoluteMaxCapacity": 100,
"absoluteUsedCapacity": 0,
"weight": -1,
"normalizedWeight": 0,
"numApplications": 0,
"maxParallelApps": 2147483647,
"queueName": "parent1",
"isAbsoluteResource": true,
"state": "RUNNING",
"queues": {
"queue": [
{
"type": "capacitySchedulerLeafQueueInfo",
"queuePath": "root.parent1.TestUser",
"capacity": 25,
"usedCapacity": 0,
"maxCapacity": 100,
"absoluteCapacity": 12.5,
"absoluteMaxCapacity": 100,
"absoluteUsedCapacity": 0,
"weight": -1,
"normalizedWeight": 0,
"numApplications": 0,
"maxParallelApps": 2147483647,
"queueName": "TestUser",
"isAbsoluteResource": false, {code}
The parent1 queue is in absolute mode, but the dynamically created TestUser
queue is not in absolute mode. BTW the cluster had 8 vCores and 8 GB of memory.
was (Author: tdomok):
*I verified that Queue ACL can't be configured via the leaf-queue-template,
indeed they are inherited from parent.*
*I verified that Absolute Resource can't be configured via the
leaf-queue-template.*
*Details on Queue ACL:*
1. Do not allow anybody to submit/admin apps on the root queue
{code:java}
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value> </value>
</property>
<property>
<name>yarn.scheduler.capacity.root.acl_administer_queue</name>
<value> </value>
</property> {code}
2. Allow only nottdomok, testuser and TestUser for the parent1 queue
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.acl_submit_applications</name>
<value>nottdomok,testuser,TestUser</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.acl_administer_queue</name>
<value>nottdomok,testuser,TestUser</value>
</property> {code}
3. Try to allow tdomok using the leaf-queue-template
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.acl_submit_applications</name>
<value>tdomok</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.acl_administer_queue</name>
<value>tdomok</value>
</property> {code}
4. Results: TestUser can submit a job to the dynamically created TestUser
queue, while the user tdomok can't submit to the tdomok queue which would be
dynamically created.
{code:java}
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException:
org.apache.hadoop.security.AccessControlException: User tdomok does not have
permission to submit application_1636622725698_0001 to queue tdomok {code}
*Details on Absolute Resource*
1. Configuration
{code:java}
<property>
<name>yarn.scheduler.capacity.root.parent1.capacity</name>
<value>[memory=4096,vcores=8]</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.capacity</name>
<value>[memory=1024,vcores=4]</value>
</property> {code}
2. curl "http://localhost:8088/ws/v1/cluster/scheduler"; --verbose | jq &>
cfg.xml
{code:java}
"queues": {
"queue": [
{
"queuePath": "root.parent1",
"capacity": 50,
"usedCapacity": 0,
"maxCapacity": 100,
"absoluteCapacity": 50,
"absoluteMaxCapacity": 100,
"absoluteUsedCapacity": 0,
"weight": -1,
"normalizedWeight": 0,
"numApplications": 0,
"maxParallelApps": 2147483647,
"queueName": "parent1",
"isAbsoluteResource": true,
"state": "RUNNING",
"queues": {
"queue": [
{
"type": "capacitySchedulerLeafQueueInfo",
"queuePath": "root.parent1.TestUser",
"capacity": 25,
"usedCapacity": 0,
"maxCapacity": 100,
"absoluteCapacity": 12.5,
"absoluteMaxCapacity": 100,
"absoluteUsedCapacity": 0,
"weight": -1,
"normalizedWeight": 0,
"numApplications": 0,
"maxParallelApps": 2147483647,
"queueName": "TestUser",
"isAbsoluteResource": false, {code}
The parent1 queue is in absolute mode, but the dynamically created TestUser
queue is not in absolute mode. BTW the cluster had 8 vCores and 8 GB of memory.
> Investigation: Verify if legacy AQC works as documented
> -------------------------------------------------------
>
> Key: YARN-10922
> URL: https://issues.apache.org/jira/browse/YARN-10922
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Szilard Nemeth
> Assignee: Tamas Domok
> Priority: Minor
> Attachments: capacity-scheduler.xml
>
>
> Quoting from the Capacity Scheduler documentation:
> https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
> Section: "Dynamic Auto-Creation and Management of Leaf Queues"
> The task is to verify if legacy AQC works like this:
> {quote}
> The parent queue which has been enabled for auto leaf queue creation,
> supports the configuration of template parameters for automatic configuration
> of the auto-created leaf queues. The auto-created queues support all of the
> leaf queue configuration parameters except for Queue ACL, Absolute Resource
> configurations. Queue ACLs are currently inherited from the parent queue i.e
> they are not configurable on the leaf queue template
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
