[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[
https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358899#comment-15358899
]
Prabhu Joseph commented on YARN-5295:
-
Hi [~sunilg], In addition, we also need to include below code snippet in
UserGroupMappingPlacementRule#getMappedQueue before returning the mapped queue,
which will return a valid queue which is a existent leaf queue.
{code}
for (QueueMapping mapping : mappings) {
if (queue == null || !(queue instanceof LeafQueue)) {
continue;
}
return queue;
}
{code}
> YARN queue-mappings to check Queue is present before submitting job
> ---
>
> Key: YARN-5295
> URL: https://issues.apache.org/jira/browse/YARN-5295
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.7.2
>Reporter: Prabhu Joseph
>
> In yarn Queue-Mappings, Yarn should check if the queue is present before
> submitting the job. If not present it should go to next mapping available.
> For example if we have
> yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform
> and I submit job with user "test" and if there is no "test" queue then it
> should check the second mapping (g:edw:platform) in the list and if test is
> part of edw group it should submit job in platform queue.
> Below Sanity checks has to be done for the mapped queue in the list and if it
> fails then the the next queue mapping has to be chosen, when there is no
> queue mapping passing the sanity check, only then the application has to be
> Rejected.
> 1. is queue present
> 2. is queue not a leaf queue
> 3. is user either have ACL Submit_Applications or Administer_Queue of the
> queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[
https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356770#comment-15356770
]
Sunil G commented on YARN-5295:
---
Hi [~Prabhu Joseph]
In YARN-3635, a new {{PlacementManager}} is added. It takes care of queue
mappings in general across schedulers.
Code snippet from {{CS#getUserGroupMappingPlacementRule}},
{code}
if (queue == null || !(queue instanceof LeafQueue)) {
throw new IOException("mapping contains invalid or non-leaf queue "
+ mappingQueue);
}
{code}
While loading queue-mappings from conf, a validation is done for *non-existent*
and *non-leafQueues*. And this {{QueueMapping}} is used in {{PlacementManager}}
to do the placement activity in RMAppManager.
But this is not present in branch2.7 against which you raised this ticket.
[~leftnoteasy], could we backport YARN-3635 to branch-2.7. Its present in
branch-2.8 already.
> YARN queue-mappings to check Queue is present before submitting job
> ---
>
> Key: YARN-5295
> URL: https://issues.apache.org/jira/browse/YARN-5295
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.7.2
>Reporter: Prabhu Joseph
>
> In yarn Queue-Mappings, Yarn should check if the queue is present before
> submitting the job. If not present it should go to next mapping available.
> For example if we have
> yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform
> and I submit job with user "test" and if there is no "test" queue then it
> should check the second mapping (g:edw:platform) in the list and if test is
> part of edw group it should submit job in platform queue.
> Below Sanity checks has to be done for the mapped queue in the list and if it
> fails then the the next queue mapping has to be chosen, when there is no
> queue mapping passing the sanity check, only then the application has to be
> Rejected.
> 1. is queue present
> 2. is queue not a leaf queue
> 3. is user either have ACL Submit_Applications or Administer_Queue of the
> queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[ https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351450#comment-15351450 ] Prabhu Joseph commented on YARN-5295: - Yes, doing Sanity Check 1 and 2 well before in getMappedQueue is suffice to help administrators to configure a default queue to any user or group in case of no valid queue mapping. For example, with this fix, Administrators can allow any new user added and who does not have queue created with same user name can still be placed in default queue through list of queue mappings u:%user:%user,u:%user:default > YARN queue-mappings to check Queue is present before submitting job > --- > > Key: YARN-5295 > URL: https://issues.apache.org/jira/browse/YARN-5295 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.7.2 >Reporter: Prabhu Joseph > > In yarn Queue-Mappings, Yarn should check if the queue is present before > submitting the job. If not present it should go to next mapping available. > For example if we have > yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform > and I submit job with user "test" and if there is no "test" queue then it > should check the second mapping (g:edw:platform) in the list and if test is > part of edw group it should submit job in platform queue. > Below Sanity checks has to be done for the mapped queue in the list and if it > fails then the the next queue mapping has to be chosen, when there is no > queue mapping passing the sanity check, only then the application has to be > Rejected. > 1. is queue present > 2. is queue not a leaf queue > 3. is user either have ACL Submit_Applications or Administer_Queue of the > queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[
https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351384#comment-15351384
]
Sunil G commented on YARN-5295:
---
Thanks [~Prabhu Joseph] for the clarification.
Yes, we select a queue and then we do sanity after that. And if sanity check
(validations) fails for this selected queue, we will reject the app submission.
In this given scenario, may be a group configuration seems valid and the jira
is focussing on that point. Got it. +1 for the approach.
Interestingly we need to see how 3rd point is to be handled {{3. is user either
have ACL Submit_Applications or Administer_Queue of the queue.}}, we may
possibly relax some ACLs here if we do that. If a user (user *test*) is not
given permission to queue (queue named *test* in this context), and same user's
group (group *edw*) has permission in another queue (queue named *platform* in
this context), I think we have to give preference for user than group. So i
think current implementation holds good for 3rd point. Thoughts?
> YARN queue-mappings to check Queue is present before submitting job
> ---
>
> Key: YARN-5295
> URL: https://issues.apache.org/jira/browse/YARN-5295
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.7.2
>Reporter: Prabhu Joseph
>
> In yarn Queue-Mappings, Yarn should check if the queue is present before
> submitting the job. If not present it should go to next mapping available.
> For example if we have
> yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform
> and I submit job with user "test" and if there is no "test" queue then it
> should check the second mapping (g:edw:platform) in the list and if test is
> part of edw group it should submit job in platform queue.
> Below Sanity checks has to be done for the mapped queue in the list and if it
> fails then the the next queue mapping has to be chosen, when there is no
> queue mapping passing the sanity check, only then the application has to be
> Rejected.
> 1. is queue present
> 2. is queue not a leaf queue
> 3. is user either have ACL Submit_Applications or Administer_Queue of the
> queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[ https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351347#comment-15351347 ] Prabhu Joseph commented on YARN-5295: - [~sunilg] Yes, if test queue is present, the application submitted by test user placed into test queue. But if test queue is not present or if test queue is not a leaf queue or if test user does not have either Submit_Applications or Administer_Queue ACL, then the application is rejected. Instead, the getMappedQueue in CapacityScheduler can do the three sanity checks and return a valid queue that is platform instead of test. (Assuming test user passes the sanity checks on platform Queue) Currently the sanity checks are done separately after deciding the queue to be placed, instead sanity checks can be included in getMappedQueue logic, where once queue mapping is chosen from the list, the sanity checks can be done and if it fails, then move to the next queue mapping in the list. > YARN queue-mappings to check Queue is present before submitting job > --- > > Key: YARN-5295 > URL: https://issues.apache.org/jira/browse/YARN-5295 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 2.7.2 >Reporter: Prabhu Joseph > > In yarn Queue-Mappings, Yarn should check if the queue is present before > submitting the job. If not present it should go to next mapping available. > For example if we have > yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform > and I submit job with user "test" and if there is no "test" queue then it > should check the second mapping (g:edw:platform) in the list and if test is > part of edw group it should submit job in platform queue. > Below Sanity checks has to be done for the mapped queue in the list and if it > fails then the the next queue mapping has to be chosen, when there is no > queue mapping passing the sanity check, only then the application has to be > Rejected. > 1. is queue present > 2. is queue not a leaf queue > 3. is user either have ACL Submit_Applications or Administer_Queue of the > queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5295) YARN queue-mappings to check Queue is present before submitting job
[
https://issues.apache.org/jira/browse/YARN-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350703#comment-15350703
]
Sunil G commented on YARN-5295:
---
Hi [~Prabhu Joseph]
Thanks for raising this issue. Couple of doubts here,
If we have {{u:%user:%user}} and {{g:edw:platform}}, and assume *test* is not
present in cluster. But *test* user is still a part of *edw* group. So we have
to submit to *platform* queue.
If *test* queue was present, this app would have got submitted to *test*
itself. Am I correct?
Being said this, we already do a set of checks in CS (I am mentioning 2.7.2 and
trunk)
In trunk, we have {{PlacementManager}} which will do the check before the app
itself is submitted to scheduler. Ref YARN-3635. Also if we use ACLs, we will
have that check also done in {{RMAppManager#submitApplication}} itself.
In 2.7.2, this check is done in CS itself. So application was added to
scheduler and then we will get an APP_REJECTED event.
So I think we do check for queue now, but we are not checking whether its a
leaf queue. And ACLS are already handled in trunk. Could you pls help to
correct if i understood the problem differently.
> YARN queue-mappings to check Queue is present before submitting job
> ---
>
> Key: YARN-5295
> URL: https://issues.apache.org/jira/browse/YARN-5295
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.7.2
>Reporter: Prabhu Joseph
>
> In yarn Queue-Mappings, Yarn should check if the queue is present before
> submitting the job. If not present it should go to next mapping available.
> For example if we have
> yarn.scheduler.capacity.queue-mappings=u:%user:%user,g:edw:platform
> and I submit job with user "test" and if there is no "test" queue then it
> should check the second mapping (g:edw:platform) in the list and if test is
> part of edw group it should submit job in platform queue.
> Below Sanity checks has to be done for the mapped queue in the list and if it
> fails then the the next queue mapping has to be chosen, when there is no
> queue mapping passing the sanity check, only then the application has to be
> Rejected.
> 1. is queue present
> 2. is queue not a leaf queue
> 3. is user either have ACL Submit_Applications or Administer_Queue of the
> queue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
