[ https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291334#comment-17291334 ]
Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 1:53 AM: ------------------------------------------------------------------ Tested the fix on trunk using a single node cluster using the following steps: * Create a _Standard_ user with a username containing a "." (dot) on Mac -> {{firstname.lastname}} as per [1]. * Set up the single node cluster for trunk and enable the following permissions such that the new user can have rwx permissions under /tmp folder as otherwise job submissions will fail: {code} admin@mac hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -chmod -R a+rwx /tmp 2021-02-26 12:21:54,034 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable {code} * Add the following setting under {{hadoop-3.4.0-SNAPSHOT/etc/hadoop/capacity-scheduler.xml}} to enable weights for the {{firstname.lastname}} user: {code} <property> <name>yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight</name> <value>0.76</value> </property> {code} * Ensure HDFS & RM services are running on the single node cluster and run the sleep job as the new user: {code} admin@mac hadoop-3.4.0-SNAPSHOT % sudo -u firstname.lastname bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar sleep -m 1 -mt 600000 <Enter admin password> 2021-02-26 12:21:57,305 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2021-02-26 12:21:57,989 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032 2021-02-26 12:21:58,101 INFO client.AHSProxy: Connecting to Application History server at /0.0.0.0:10200 2021-02-26 12:21:58,581 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/firstname.lastname/.staging/job_1614302477419_0001 2021-02-26 12:21:59,343 INFO mapreduce.JobSubmitter: number of splits:1 2021-02-26 12:21:59,440 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1614302477419_0001 2021-02-26 12:21:59,440 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2021-02-26 12:21:59,573 INFO conf.Configuration: resource-types.xml not found 2021-02-26 12:21:59,574 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2021-02-26 12:21:59,987 INFO impl.YarnClientImpl: Submitted application application_1614302477419_0001 2021-02-26 12:22:00,025 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1614302477419_0001/ 2021-02-26 12:22:00,025 INFO mapreduce.Job: Running job: job_1614302477419_0001 2021-02-26 12:22:07,128 INFO mapreduce.Job: Job job_1614302477419_0001 running in uber mode : false 2021-02-26 12:22:07,130 INFO mapreduce.Job: map 0% reduce 0% ... {code} * Check the "_Active Users Info_" section after expanding the {{root.default}} queue on the RM Scheduler page at http://localhost:8088/cluster/scheduler. It should contain 0.76 instead of 1.0. Confirmed this to be working after the change. [1] https://support.apple.com/en-au/guide/mac-help/mtusr001/mac JUnit has also been updated to ensure that the weights for usernames containing a dot are set up accordingly. Meanwhile, also fixed the junit to ensure that that the {{assertEquals}} with float arguments are picked up instead of double by appending the suffix "f" to the literal values (see [2] for more info) and also removed un-necessary unboxing using {{floatValue()}} as this is not required. [2] https://stackoverflow.com/questions/3033137/representing-float-values-in-java was (Author: sahuja): Tested the fix on trunk using a single node cluster using the following steps: * Create a _Standard_ user with a username containing a "." (dot) on Mac -> {{firstname.lastname}} as per [1]. * Set up the single node cluster for trunk and enable the following permissions such that the new user can have rwx permissions under /tmp folder as otherwise job submissions will fail: {code} admin@mac hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -chmod -R a+rwx /tmp 2021-02-26 12:21:54,034 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable {code} * Add the following setting under {{hadoop-3.4.0-SNAPSHOT/etc/hadoop/capacity-scheduler.xml}} to enable weights for the {{firstname.lastname}} user: {code} <property> <name>yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight</name> <value>0.76</value> </property> {code} * Ensure HDFS & RM services are running on the single node cluster and run the sleep job as the new user: {code} admin@mac hadoop-3.4.0-SNAPSHOT % sudo -u firstname.lastname bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-SNAPSHOT-tests.jar sleep -m 1 -mt 600000 <Enter admin password> 2021-02-26 12:21:57,305 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2021-02-26 12:21:57,989 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032 2021-02-26 12:21:58,101 INFO client.AHSProxy: Connecting to Application History server at /0.0.0.0:10200 2021-02-26 12:21:58,581 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/firstname.lastname/.staging/job_1614302477419_0001 2021-02-26 12:21:59,343 INFO mapreduce.JobSubmitter: number of splits:1 2021-02-26 12:21:59,440 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1614302477419_0001 2021-02-26 12:21:59,440 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2021-02-26 12:21:59,573 INFO conf.Configuration: resource-types.xml not found 2021-02-26 12:21:59,574 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2021-02-26 12:21:59,987 INFO impl.YarnClientImpl: Submitted application application_1614302477419_0001 2021-02-26 12:22:00,025 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1614302477419_0001/ 2021-02-26 12:22:00,025 INFO mapreduce.Job: Running job: job_1614302477419_0001 2021-02-26 12:22:07,128 INFO mapreduce.Job: Job job_1614302477419_0001 running in uber mode : false 2021-02-26 12:22:07,130 INFO mapreduce.Job: map 0% reduce 0% ... {code} * Check the "_Active Users Info_" section after expanding the {{root.default}} queue on the RM Scheduler page at http://localhost:8088/cluster/scheduler. It should contain 0.76 instead of 1.0. Confirmed this to be working after the change. [1] https://support.apple.com/en-au/guide/mac-help/mtusr001/mac JUnit has also been updated to ensure that the weights for usernames containing a dot are set up accordingly. Meanwhile, also fixed the junit to ensure that that the {{assertEquals}} with float arguments are picked up instead of double by appending the suffix "f" to the literal values and also removed un-necessary unboxing using {{floatValue()}} as this is not required. > Capacity Scheduler fails to handle user weights for a user that has a "." > (dot) in it > ------------------------------------------------------------------------------------- > > Key: YARN-10652 > URL: https://issues.apache.org/jira/browse/YARN-10652 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Affects Versions: 3.3.0 > Reporter: Siddharth Ahuja > Assignee: Siddharth Ahuja > Priority: Major > Attachments: Correct user weight of 0.76 picked up for the user with > a dot after the patch.png, Incorrect default user weight of 1.0 being picked > for the user with a dot before the patch.png, YARN-10652.001.patch > > > AD usernames can have a "." (dot) in them i.e. they can be of the format -> > {{firstname.lastname}}. However, if you specify a username with this format > against the Capacity Scheduler setting -> > {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}}, > it fails to be applied and is instead assigned the default of 1.0f weight. > This renders the user weight feature (being used as a means of setting user > priorities for a queue) unusable for such users. > This limitation comes from [1]. From [1], only word characters (A word > character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no > good for AD names that contain a "." (dot). > Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and > HADOOP-15395 and the outcome was to use non-whitespace characters i.e. > instead of {{\w+}}, use {{\S+}}. > We could go down similar path and unblock this feature for the AD usernames > with a "." (dot) in them. > [1] > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953 > [2] > https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org