[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264903#comment-17264903
]
Peter Bacsko commented on YARN-10506:
-------------------------------------
Hi guys. I haven't really contributed to this patch and there's some big
discussion going on here, but I'm adding my 2 cents.
To give you some background: I introduced the "create" flag for the new mapping
rules, simply because FS has the same - auto-creation depends on the rule and
users who migrate from FS will find this intuitive, even if the rule format has
changed drastically. Now, I was under the impression that with weight mode and
AQC, we're trying to approximate FS behaviour as much as possible, because the
whole "project" came up in the context on FS->CS migration.
So I thought that in this mode, users are allowed to create queues anywhere in
the hierarchy, eg. with {{<specified>}} rule, you can definitely do this in FS,
there is no restriction, at least I'm not aware of it. The FS property
"allow-undeclared-pools" is translated internally to {{<specified>}} rule.
Again, the was the idea.
On the other hand, it's also true that if we go this way, legacy percentage
mode and weight mode will diverge significantly, so the behavior will be
totally different. In pct/absolute mode, you can explicitly say where queues
can be created, but if a user switches to weight mode, they cannot do this,
which might be surprising for them (or maybe not - this is a pure speculation
at this point).
If we really want to limit users where they can create queues (as I can see
this is clearly the goal here) then this leads us to the problem what Gergely
just discussed above, have a "create" flag in the rule and have a similar
property which restricts whether a queue can be created, this can lead to some
unexpected behavior, especially if their priority is reversed. At the very
least, legacy FS users will be confused like hell IMO.
We also have to think about the following scenario: a user who just migrated
from FS, still wants the exact FS auto-queue semantics and uses
"allow-undeclared-pools" or a {{<specified>}} rule. So if that's the case, we
have to be able to set this up easily. That is, if they have 200 queues and
they have to set this property for _every single queue_, I can easily imagine
that their head will explode. So we should consider having the property
"auto-queue-creation-v2.enabled" as a setting which is automatically inherited
by all child queues, recursively. Or choose the default value to be "true"
which is probably the simplest thing to do.
My conclusion is:
1) I think we have to keep the "create" flag on a rule level if we want FS
rules to work the same way. Some rules might have create=true, some of them
might not, so we can't rely on having the queue properties alone.
2) If we want to introduce "auto-queue-creation-v2.enabled", it's important to
think about whether to make it inheritable or not. If it's not, then this can
potentially cause a huge pain for some users. Alternative approach: have a
separate property which applies to the whole hierarchy globally, which can be
overridden on a per-queue basis, this is a very common approach in CS.
3) I very much agree with this statement: _"Mapping rules will return a path
based on their configuration, but I don't think the placement rule should be
overriding the CS configuration"_. If we truly want both, then the queue
property should have its final say about the creation. The rule says "let's
create this queue if it doesn't exist", but then we must check
"auto-queue-creation-v2.enabled" for the target path to see if the queue can be
created at the given point in the hierarchy. I should not be the other way
around.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
