[ https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292620#comment-17292620 ]
Wilfred Spiegelenburg commented on YARN-10652: ---------------------------------------------- I do not see the relation with placement rules or the FS for this fix at all. The weight of a queue can be used with or without a placement rule. It is a setting on a queue. Having this setting on a user based queue also does not really make sense. It gives more resources to one user over others in the queue. So I can see this being used on a parent queue or a group queue but not on an individuals leaf queue. On top of that the queue path of the configuration setting is not affected by this change. We're talking about this setting: {code:java} yarn.scheduler.capacity.<queue-path>.user-settings.<user-name>.weight{code} The weight is retrieved for a specific queue as defined in the _<queue-path>._ That part is already resolved and is not changed. The only resolution that is changed is the _<user-name>_ part between the words _user-settings_. and _.weight._ The queue path could be anything and is not in play here. It could even a fixed configured queue or one mapped on a group name. The administrator should know as minimal as possible, preferably nothing, about the internals for storing users in the CS. If the queue mapping rule for the user changes the dots to make it a single part of the queue path then that is independent of this change. It still does not change the way the user is stored in the CS. It changes the way you map a user to a queue in the placement rules. On the FS side we thought about standardising dot usage. We considered both cases using and not _dot_ in the config files in user names. When I looked at it I was not sure which was the correct solution. It could lead to strange behaviour and extra administrative work. The admin forgets to remove the dot and all of a sudden the config does not apply. That is why it never went further than just the Jira YARN-5674. With this change as proposed you will support weights for all users with a dot except for the user called: something_.weights_ That will be the only user set that breaks which is far less than breaking all users with a dot in the username. I do not see any other <user-name> bound properties in the configuration at the moment. If you want to solve the generic dot issue for user based placement then that is outside of this change. > Capacity Scheduler fails to handle user weights for a user that has a "." > (dot) in it > ------------------------------------------------------------------------------------- > > Key: YARN-10652 > URL: https://issues.apache.org/jira/browse/YARN-10652 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Affects Versions: 3.3.0 > Reporter: Siddharth Ahuja > Assignee: Siddharth Ahuja > Priority: Major > Attachments: Correct user weight of 0.76 picked up for the user with > a dot after the patch.png, Incorrect default user weight of 1.0 being picked > for the user with a dot before the patch.png, YARN-10652.001.patch > > > AD usernames can have a "." (dot) in them i.e. they can be of the format -> > {{firstname.lastname}}. However, if you specify a username with this format > against the Capacity Scheduler setting -> > {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}}, > it fails to be applied and is instead assigned the default of 1.0f weight. > This renders the user weight feature (being used as a means of setting user > priorities for a queue) unusable for such users. > This limitation comes from [1]. From [1], only word characters (A word > character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no > good for AD names that contain a "." (dot). > Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and > HADOOP-15395 and the outcome was to use non-whitespace characters i.e. > instead of {{\w+}}, use {{\S+}}. > We could go down similar path and unblock this feature for the AD usernames > with a "." (dot) in them. > [1] > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953 > [2] > https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org