[jira] [Created] (YARN-10657) We should make max application per queue to support node label.

[Qi Zhu (Jira)]

Qi Zhu created YARN-10657:
-

 Summary: We should make max application per queue to support node 
label.
 Key: YARN-10657
 URL: https://issues.apache.org/jira/browse/YARN-10657
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Qi Zhu
Assignee: Qi Zhu


https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708

As we discussed in above comment:

We should deep into the label related max applications per queue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10641) Refactor the max app related update, and fix maxApllications update error when add new queues.

[Qi Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292017#comment-17292017
 ] 

Qi Zhu commented on YARN-10641:
---

Thanks [~gandras] for patient review.:D

As i added in the todo things,  i think it's a big things for us to investigate 
and discuss, i created a new Jira to handle this.

 

> Refactor the max app related update, and fix maxApllications update error 
> when add new queues.
> --
>
> Key: YARN-10641
> URL: https://issues.apache.org/jira/browse/YARN-10641
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Attachments: YARN-10641.001.patch, YARN-10641.002.patch, 
> YARN-10641.003.patch, YARN-10641.004.patch, 
> image-2021-02-20-15-49-58-677.png, image-2021-02-20-15-53-51-099.png, 
> image-2021-02-20-15-55-44-780.png, image-2021-02-20-16-29-18-519.png, 
> image-2021-02-20-16-31-13-714.png
>
>
> When refactor the update logic in YARN-10504 .
> The update max applications based abs/cap is wrong, this should be fixed, 
> because the max applications is key part to limit applications in CS.
> For example: 
> When adding a dynamic queue, the other children's max app of parent queue are 
> not updated correctly:
> !image-2021-02-20-15-53-51-099.png|width=639,height=509!  
> The new added queue's max app will updated correctly:
> !image-2021-02-20-15-55-44-780.png|width=542,height=426!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

[Qi Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292016#comment-17292016
 ] 

Qi Zhu commented on YARN-10532:
---

Thanks  [~pbacsko] for patient review.:D

Waiting for [~gandras] [~pbacsko] to final check.

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> YARN-10532.021.patch, YARN-10532.022.patch, YARN-10532.023.patch, 
> YARN-10532.024.patch, image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291919#comment-17291919
 ] 

Peter Bacsko commented on YARN-10627:
-

Thanks for the patch [~bteke] and [~zhuqi] / [~gandras] for the review.

Committed to trunk.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + 

[jira] [Updated] (YARN-10653) Fixed the findbugs issues introduced by YARN-10647.

[Eric Badger (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-10653:
---
Fix Version/s: 3.2.3
   3.1.5
   3.3.1
   3.4.0

> Fixed the findbugs issues introduced by YARN-10647.
> ---
>
> Key: YARN-10653
> URL: https://issues.apache.org/jira/browse/YARN-10653
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1, 3.1.5, 3.2.3
>
> Attachments: YARN-10653.001.patch, YARN-10653.002.patch, 
> image-2021-02-26-13-49-18-241.png
>
>
> In YARN-10647
> I fixed TestRMNodeLabelsManager failed after YARN-10501.
> But the finding bugs should be fixed also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10653) Fixed the findbugs issues introduced by YARN-10647.

[Eric Badger (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-10653:
---

Thanks for the patch, [~zhuqi], and apologies again for my mistake on the 
previous JIRA.

I have committed this to trunk (3.4), branch-3.3, branch-3.2, and branch-3.1

> Fixed the findbugs issues introduced by YARN-10647.
> ---
>
> Key: YARN-10653
> URL: https://issues.apache.org/jira/browse/YARN-10653
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10653.001.patch, YARN-10653.002.patch, 
> image-2021-02-26-13-49-18-241.png
>
>
> In YARN-10647
> I fixed TestRMNodeLabelsManager failed after YARN-10501.
> But the finding bugs should be fixed also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

[Hadoop QA (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291897#comment-17291897
 ] 

Hadoop QA commented on YARN-10627:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m  3s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 146 unchanged - 16 fixed = 146 total (was 162) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 50s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubun

[jira] [Commented] (YARN-10640) Adjust the queue Configured capacity to Configured weight number for weight mode in UI.

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291894#comment-17291894
 ] 

Peter Bacsko commented on YARN-10640:
-

[~zhuqi] thanks for the patch, it's useful.

There is one thing I was thinking: in weight mode, we configure weights but 
inside CS we still calculate percentages. So how is it possible that 
"Configured capacity" is 0/0? For example, percentage values are properly 
shown, those are not 0.

[~gandras] [~bteke] do you know why {{}} is printed even 
for static queues?


> Adjust the queue Configured capacity to  Configured weight number for weight 
> mode in UI.
> 
>
> Key: YARN-10640
> URL: https://issues.apache.org/jira/browse/YARN-10640
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10640.001.patch, YARN-10640.002.patch, 
> image-2021-02-20-11-21-50-306.png, image-2021-02-20-14-18-56-261.png, 
> image-2021-02-20-14-19-30-767.png
>
>
> In weight mode:
> Both the weight mode static queue and the dynamic queue will show the 
> Configured Capacity to 0. I think this should change to Configured Weight if 
> we use weight mode, this will be helpful.
> Such as in dynamic weight mode queue:
> !image-2021-02-20-11-21-50-306.png|width=528,height=374!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291889#comment-17291889
 ] 

Peter Bacsko commented on YARN-10532:
-

[~zhuqi] I think it's very good now. Maybe I'll check another round on Monday 
but this one looks good enough to get committed.

[~gandras] [~bteke] you guys pls do a final review.

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> YARN-10532.021.patch, YARN-10532.022.patch, YARN-10532.023.patch, 
> YARN-10532.024.patch, image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10656) Parsing error in CapacityScheduler.md

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291886#comment-17291886
 ] 

Peter Bacsko commented on YARN-10656:
-

Thanks [~aajisaka] for noticing this. I should have tried it before committing.

+1.

> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> mvn site failed: 
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
> hadoop-yarn-site: Error parsing 
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
>  line [-1] Error parsing the model: Unable to execute macro in the document: 
> toc -> [Help 1]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10653) Fixed the findbugs issues introduced by YARN-10647.

[Eric Badger (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291877#comment-17291877
 ] 

Eric Badger commented on YARN-10653:


Ahhh ok I see it now. The findbugs error is in "trunk Compile Tests" while the 
"patch Compile Tests" findbugs comes back clean. Ok cool. +1 on the patch. 
Committing patch 001 now

> Fixed the findbugs issues introduced by YARN-10647.
> ---
>
> Key: YARN-10653
> URL: https://issues.apache.org/jira/browse/YARN-10653
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10653.001.patch, YARN-10653.002.patch, 
> image-2021-02-26-13-49-18-241.png
>
>
> In YARN-10647
> I fixed TestRMNodeLabelsManager failed after YARN-10501.
> But the finding bugs should be fixed also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10656) Parsing error in CapacityScheduler.md

[Akira Ajisaka (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291850#comment-17291850
 ] 

Akira Ajisaka commented on YARN-10656:
--

Hi [~pbacsko], would you check https://github.com/apache/hadoop/pull/2725 ?

> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> mvn site failed: 
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
> hadoop-yarn-site: Error parsing 
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
>  line [-1] Error parsing the model: Unable to execute macro in the document: 
> toc -> [Help 1]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10656) Parsing error in CapacityScheduler.md

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reassigned YARN-10656:


Assignee: Akira Ajisaka

> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> mvn site failed: 
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
> hadoop-yarn-site: Error parsing 
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
>  line [-1] Error parsing the model: Unable to execute macro in the document: 
> toc -> [Help 1]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10656) Parsing error in CapacityScheduler.md

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated YARN-10656:
--
Labels: pull-request-available  (was: )

> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> mvn site failed: 
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
> hadoop-yarn-site: Error parsing 
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
>  line [-1] Error parsing the model: Unable to execute macro in the document: 
> toc -> [Help 1]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10656) Parsing error in CapacityScheduler.md

[Akira Ajisaka (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291846#comment-17291846
 ] 

Akira Ajisaka commented on YARN-10656:
--

How to reproduce:
{noformat}
$ ./start-build-env.sh
$ mvn clean -pl hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
# got an error
$ mvn site -pl hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
# try again. somehow it worked
$ mvn site -pl hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
{noformat}

I got the detailed error message by upgrading maven-site-plugin to 3.9.1.
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-site-plugin:3.9.1:site (default-site) on project 
hadoop-yarn-site: Error parsing 
'/home/aajisaka/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
 line [14] Error parsing the model: Unable to execute macro in the document: 
toc (position: COMMENT seen ...: Capacity Scheduler... @14:77) caused by: 
org.apache.maven.doxia.macro.MacroExecutionException: ParseException: Error 
parsing the model: end tag name  must match start tag name  
from line 79 (position: TEXT seen ...s they should be able to use more 
resources than calculated. ... @79:3003)  -> [Help 1]
{noformat}

> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Priority: Major
>
> mvn site failed: 
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
> hadoop-yarn-site: Error parsing 
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
>  line [-1] Error parsing the model: Unable to execute macro in the document: 
> toc -> [Help 1]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10656) Parsing error in CapacityScheduler.md

[Akira Ajisaka (Jira)]

Akira Ajisaka created YARN-10656:


 Summary: Parsing error in CapacityScheduler.md
 Key: YARN-10656
 URL: https://issues.apache.org/jira/browse/YARN-10656
 Project: Hadoop YARN
  Issue Type: Bug
  Components: documentation
Reporter: Akira Ajisaka


mvn site failed: 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
hadoop-yarn-site: Error parsing 
'/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
 line [-1] Error parsing the model: Unable to execute macro in the document: 
toc -> [Help 1]
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

[Benjamin Teke (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.006.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

[Benjamin Teke (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291759#comment-17291759
 ] 

Benjamin Teke commented on YARN-10627:
--

[~pbacsko] sure, fixed it.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= curren

[jira] [Comment Edited] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291728#comment-17291728
 ] 

Peter Bacsko edited comment on YARN-10623 at 2/26/21, 4:10 PM:
---

Thanks [~zhuqi] for the update. I missed a couple of things in my previous 
review.

1.
{noformat}
clock = SystemClock.getInstance();
{noformat}

Use {{MonotonicClock}}. Looking at the comments of {{SystemClock}}, that class 
is a better choice.

2. {{waitFor}} usage can be simplified:

{noformat}
GenericTestUtils.waitFor(() ->
  cs.getConfiguration().getMaximumSystemApplications() != 1,
500L, 10_000L);
{noformat}

I think 100ms interval time is a bit small, 500ms seems to be a better 
compromise. Watch out for the line length or checkstyle will complain.

3. 
{noformat}
configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");

configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");
{noformat}

You can also use 
{{FileSystemBasedConfigurationProvider.class.getCanonicalName()}} here.

4. 
{noformat}

csConf.set(CapacitySchedulerConfiguration.MAXIMUM_SYSTEM_APPLICATIONS,
"5000");
{noformat}

For consistency reasons, I suggest using {{setInt()}} here.

5.
{noformat}
  @VisibleForTesting
  public long getLastReloadAttempt() {
return lastReloadAttempt;
  }

  @VisibleForTesting
  public long getLastModified() {
return lastModified;
  }

  @VisibleForTesting
  public Clock getClock() {
return clock;
  }

  @VisibleForTesting
  public boolean getLastReloadAttemptFailed() {
return  lastReloadAttemptFailed;
  }
{noformat}

As I can see, these methods are only called from test cases. You can reduce the 
visibility to package private, so just remove "public".

6. Note that stack traces are still not visible in the latest patch:
{noformat}
LOG.error("Can't refresh queue: " + e.getMessage());
...
LOG.error("Can't get file status for refresh : " + e.getMessage());
{noformat}

This is preferred:
{noformat}
LOG.error("Can't refresh queue", e);
...
LOG.error("Can't get file status for refresh", e);
{noformat}


was (Author: pbacsko):
Thanks [~zhuqi] for the update. I missed a couple of things in my previous 
review.

1.
{noformat}
clock = SystemClock.getInstance();
{noformat}

I suggest using {{MonotonicClock}}. Looking at the comments of {{SystemClock}}, 
that class is a better choice.

2. {{waitFor}} usage can be simplified:

{noformat}
GenericTestUtils.waitFor(() ->
  cs.getConfiguration().getMaximumSystemApplications() != 1,
500L, 10_000L);
{noformat}

I think 100ms interval time is a bit small, 500ms seems to be a better 
compromise. Watch out for the line length or checkstyle will complain.

3. 
{noformat}
configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");

configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");
{noformat}

You can also use 
{{FileSystemBasedConfigurationProvider.class.getCanonicalName()}} here.

4. 
{noformat}

csConf.set(CapacitySchedulerConfiguration.MAXIMUM_SYSTEM_APPLICATIONS,
"5000");
{noformat}

For consistency reasons, I suggest using {{setInt()}} here.

5.
{noformat}
  @VisibleForTesting
  public long getLastReloadAttempt() {
return lastReloadAttempt;
  }

  @VisibleForTesting
  public long getLastModified() {
return lastModified;
  }

  @VisibleForTesting
  public Clock getClock() {
return clock;
  }

  @VisibleForTesting
  public boolean getLastReloadAttemptFailed() {
return  lastReloadAttemptFailed;
  }
{noformat}

As I can see, these methods are only called from test cases. You can reduce the 
visibility to package private, so just remove "public".

6. Note that stack traces are still not visible in the latest patch:
{noformat}
LOG.error("Can't refresh queue: " + e.getMessage());
...
LOG.error("Can't get file status for refresh : " + e.getMessage());
{noformat}

This is preferred:
{noformat}
LOG.error("Can't refresh queue", e);
...
LOG.error("Can't get file status for refresh", e);
{noformat}

> Capacity scheduler should support refresh queue automatically by a thread 
> policy.
> --

[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291728#comment-17291728
 ] 

Peter Bacsko commented on YARN-10623:
-

Thanks [~zhuqi] for the update. I missed a couple of things in my previous 
review.

1.
{noformat}
clock = SystemClock.getInstance();
{noformat}

I suggest using {{MonotonicClock}}. Looking at the comments of {{SystemClock}}, 
that class is a better choice.

2. {{waitFor}} usage can be simplified:

{noformat}
GenericTestUtils.waitFor(() ->
  cs.getConfiguration().getMaximumSystemApplications() != 1,
500L, 10_000L);
{noformat}

I think 100ms interval time is a bit small, 500ms seems to be a better 
compromise. Watch out for the line length or checkstyle will complain.

3. 
{noformat}
configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");

configuration.set(YarnConfiguration.RM_CONFIGURATION_PROVIDER_CLASS,
"org.apache.hadoop.yarn.FileSystemBasedConfigurationProvider");
{noformat}

You can also use 
{{FileSystemBasedConfigurationProvider.class.getCanonicalName()}} here.

4. 
{noformat}

csConf.set(CapacitySchedulerConfiguration.MAXIMUM_SYSTEM_APPLICATIONS,
"5000");
{noformat}

For consistency reasons, I suggest using {{setInt()}} here.

5.
{noformat}
  @VisibleForTesting
  public long getLastReloadAttempt() {
return lastReloadAttempt;
  }

  @VisibleForTesting
  public long getLastModified() {
return lastModified;
  }

  @VisibleForTesting
  public Clock getClock() {
return clock;
  }

  @VisibleForTesting
  public boolean getLastReloadAttemptFailed() {
return  lastReloadAttemptFailed;
  }
{noformat}

As I can see, these methods are only called from test cases. You can reduce the 
visibility to package private, so just remove "public".

6. Note that stack traces are still not visible in the latest patch:
{noformat}
LOG.error("Can't refresh queue: " + e.getMessage());
...
LOG.error("Can't get file status for refresh : " + e.getMessage());
{noformat}

This is preferred:
{noformat}
LOG.error("Can't refresh queue", e);
...
LOG.error("Can't get file status for refresh", e);
{noformat}

> Capacity scheduler should support refresh queue automatically by a thread 
> policy.
> -
>
> Key: YARN-10623
> URL: https://issues.apache.org/jira/browse/YARN-10623
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10623.001.patch, YARN-10623.002.patch, 
> YARN-10623.003.patch, YARN-10623.004.patch, YARN-10623.005.patch
>
>
> In fair scheduler, it is supported that refresh queue related conf 
> automatically by a thread to reload, but in capacity scheduler we only 
> support to refresh queue related changes by refreshQueues, it is needed for 
> our cluster to realize queue manage.
> cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras]  [~bteke] [~shuzirra]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291718#comment-17291718
 ] 

Peter Bacsko commented on YARN-10627:
-

Thanks for the patch [~bteke], could you take care of those checkstyle stuff? 
Thanks.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numCont

[jira] [Commented] (YARN-10641) Refactor the max app related update, and fix maxApllications update error when add new queues.

[Andras Gyori (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291708#comment-17291708
 ] 

Andras Gyori commented on YARN-10641:
-

Hi [~zhuqi], thank you for the patch. I always like the idea of making the code 
cleaner and fixing bugs in the meanwhile. The patch looks good to me, I have 
only minor additions:
 * I think testLeafQueueMaxAppUpdateWhenAutoCreation should be named 
testAutoQueueCreationMaxAppUpdate (this is the convention other methods are 
using)
 * The assertEquals method is used the other way around. The first value is the 
expected value, the second is the actual.
 * LeafQueue has a TODO comment with a typo.

One thing I realised while reading your comment is that we are indeed ignoring 
labeling while setting the maximum applications in a queue. As I see it, we 
overwrite the maximumApplications value each time while iterating through 
labels, which effectively makes the last label to dominate in the calculation. 
Perhaps its worth investigating a bit.

> Refactor the max app related update, and fix maxApllications update error 
> when add new queues.
> --
>
> Key: YARN-10641
> URL: https://issues.apache.org/jira/browse/YARN-10641
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Attachments: YARN-10641.001.patch, YARN-10641.002.patch, 
> YARN-10641.003.patch, YARN-10641.004.patch, 
> image-2021-02-20-15-49-58-677.png, image-2021-02-20-15-53-51-099.png, 
> image-2021-02-20-15-55-44-780.png, image-2021-02-20-16-29-18-519.png, 
> image-2021-02-20-16-31-13-714.png
>
>
> When refactor the update logic in YARN-10504 .
> The update max applications based abs/cap is wrong, this should be fixed, 
> because the max applications is key part to limit applications in CS.
> For example: 
> When adding a dynamic queue, the other children's max app of parent queue are 
> not updated correctly:
> !image-2021-02-20-15-53-51-099.png|width=639,height=509!  
> The new added queue's max app will updated correctly:
> !image-2021-02-20-15-55-44-780.png|width=542,height=426!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Andras Gyori (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291657#comment-17291657
 ] 

Andras Gyori commented on YARN-10652:
-

When I first saw this issue, I implicitly thought about the same things 
[~shuzirra] mentioned. Currently, there is no consensus how configuration 
parsing is implemented. I agree, that users should not be aware of this, but 
they might be affected nonetheless, because
 * With this fix, weight setting is working with usernames including dot
 * Other properties might not

Currently, a somewhat centralised access point is getUserPrefix, which supports 
username with dots, but as this case shows us, there might be hidden 
edge-cases, where it is still not working. One problematic point that comes to 
my mind is getPropsWithPrefix method, which is sometimes used in the 
Configuration class. This way, you might get properties for user "a", when you 
set a property for user "a.b".
That being said, this might take a lot of time to investigate, and it is better 
to fix an actual discovered case, and be vigilant about this scenario in the 
future.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10655) Limit queue creation depth relative to its first static parent

[Andras Gyori (Jira)]

Andras Gyori created YARN-10655:
---

 Summary: Limit queue creation depth relative to its first static 
parent
 Key: YARN-10655
 URL: https://issues.apache.org/jira/browse/YARN-10655
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn
Reporter: Andras Gyori
Assignee: Andras Gyori


YARN-10506 introduced a limit on the maximum depth of auto queue creation. 
This, however, only limits the levels of queue path relative to its first 
existing parent queue. It poses an unnecessary limitation on users, while 
providing no real safety net over rogue users (especially when YARN-10632 makes 
this limit configurable), because it could be incrementally circumvented by 
creating a new queue under an existing dynamic parent queue. By bounding this 
limit to the first static parent queue in the hierarchy, we could have a safer 
alternative.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291628#comment-17291628
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 1:01 PM:
--

Thanks [~pbacsko] for confirming that this issue has no direct relation to 
placement.

Thanks [~shuzirra] for your insights. I see what you are trying to say, 
however, I don't believe we need to wait for a CS-wide solution on how 
usernames are internally stored. My arguments are below:

In regards to:

{quote}
If we don't handle user names with dots properly, then we cannot say we support 
user names with dots. 
{quote}

But we are supporting usernames with dots today. Users with dots in their 
usernames can submit jobs to the cluster having CS with no issues today (again, 
I am not talking about queue placement with queues with dots here).  There are 
no errors reported when users with dots are supplied against 
"yarn.scheduler.capacity..user-settings..weight setting" 
and in fact, there should NOT be any errors when it is done so. These are 
real-world usernames and we will have to accept them from any interface, 
whether it be UI or CLI or anything. There is no need to wait to open this up 
on the front-end as they are already being stored as a String in our code, 
please see 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3902
 & 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3903.
 In regards to mapping rules, users will again specify them with real-world 
usernames. If the real-world username has a dot in them, then, thats how it 
should be accepted. How we store it internally should not matter to the user 
when they are specifying these settings.

Opening up this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should have no bearing on how this is actually stored internally in YARN CS 
whether now or in future especially when the specification of this setting is 
fairly limited to only one use and that is to specify a user's share in a 
particular queue, that's it. It does not interfere with any placement rules or 
anything else.

My solution is catering for an issue that only surfaces on the front-end, not 
the back-end so I still don't probably see how this needs to wait for any 
future refactoring on implementation side of things.



was (Author: sahuja):
Thanks [~pbacsko] for confirming that this issue has no direct relation to 
placement.

Thanks [~shuzirra] for your insights. I see what you are trying to say, 
however, I don't believe we need to wait for a CS-wide solution on how 
usernames are internally stored. My arguments are below:

In regards to:

{quote}
If we don't handle user names with dots properly, then we cannot say we support 
user names with dots. 
{quote}

But we are supporting usernames with dots today. Users with dots in their 
usernames can submit jobs to the cluster having CS with no issues today (again, 
I am not talking about queue placement with queues with dots here).  There are 
no errors reported when users with dots are supplied against 
"yarn.scheduler.capacity..user-settings..weight setting" 
and in fact, there should NOT be any errors when it is done so. These are 
real-world usernames and we will have to accept them from any interface, 
whether it be UI or CLI or anything. There is no need to wait to open this up 
on the front-end as they are already being stored as a String in our code, 
please see 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3902
 & 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3903.
 In regards to mapping rules, users will again specify them with real-world 
usernames. If the real-world username has a dot in them, then, thats how it 
should be accepted. How we store it internally should not matter to the user 
when they are specifying these settings.

Opening up this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should have no bearing on how this is actually stored internally in YARN CS 
whether now or in future.

My solution is catering for an issue that only surfaces on the front-end, not 
the back-end so I still don't probably see how this needs to wait for any 
future refactoring on implementation side of things.


> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291628#comment-17291628
 ] 

Siddharth Ahuja commented on YARN-10652:


Thanks [~pbacsko] for confirming that this issue has no direct relation to 
placement.

Thanks [~shuzirra] for your insights. I see what you are trying to say, 
however, I don't believe we need to wait for a CS-wide solution on how 
usernames are internally stored. My arguments are below:

In regards to:

{quote}
If we don't handle user names with dots properly, then we cannot say we support 
user names with dots. 
{quote}

But we are supporting usernames with dots today. Users with dots in their 
usernames can submit jobs to the cluster having CS with no issues today (again, 
I am not talking about queue placement with queues with dots here).  There are 
no errors reported when users with dots are supplied against 
"yarn.scheduler.capacity..user-settings..weight setting" 
and in fact, there should NOT be any errors when it is done so. These are 
real-world usernames and we will have to accept them from any interface, 
whether it be UI or CLI or anything. There is no need to wait to open this up 
on the front-end as they are already being stored as a String in our code, 
please see 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3902
 & 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L3903.
 In regards to mapping rules, users will again specify them with real-world 
usernames. If the real-world username has a dot in them, then, thats how it 
should be accepted. How we store it internally should not matter to the user 
when they are specifying these settings.

Opening up this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should have no bearing on how this is actually stored internally in YARN CS 
whether now or in future.

My solution is catering for an issue that only surfaces on the front-end, not 
the back-end so I still don't probably see how this needs to wait for any 
future refactoring on implementation side of things.


> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Gergely Pollak (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291614#comment-17291614
 ] 

Gergely Pollak commented on YARN-10652:
---

The problem here is the consistency. There are multiple places where user names 
are used in queue path parts, mapping rules is one example. If we don't handle 
user names with dots properly, then we cannot say we support user names with 
dots. We need to handle in each and every case, which means we need a more 
centralized or at least consistent solution. Simply replacing the dots with 
underscores for this property won't solve the issue CS wide, and we need a CS 
wide solution, which might consist of multiple separate smaller solutions like 
this, but we need to centralize this effort.

For example as soon someone introduces a rule like root.user.%user, we already 
have an issue with user names with dots, and if we handle them differently 
here, than your solution, we might get inconsistent configuration and behavior. 
Also we need to check other places where usernames with dots (and group names) 
can cause issues.

Also I prefer the FS solution for substitution with '_dot_' rather than a 
simple '_' since there is a much smaller chance of user name collision this way.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291612#comment-17291612
 ] 

Peter Bacsko edited comment on YARN-10652 at 2/26/21, 12:26 PM:


[~sahuja] you're right in saying that it has no direct relation to the 
placement.

In the first part of my comment, I was just thinking out loud that MAYBE using 
"_" instead of "." in the property is also a solution, but it comes with its 
own problems.

The placement stuff is different, it's something that we haven't considered so 
far. Currently, the new placement engine simply replaces placeholders like 
"root.users.%user" to "root.users.firstname.lastname", which is likely not what 
we want. It will not work in percentage mode, because "firstname" is a parent 
and you can't create parents under a ManagedParentQueue. In the new weight 
mode, it can work, but again, the intention is to have something like 
"root.users.firstname_lastname", just a single leaf.


was (Author: pbacsko):
[~sahuja] you're right in saying that it has no direct relation in the 
placement.

In the first part of my comment, I was just thinking out loud that MAYBE using 
"_" instead of "." in the property is also a solution, but it comes with its 
own problems.

The placement stuff is different, it's something that we haven't considered so 
far. Currently, the new placement engine simply replaces placeholders like 
"root.users.%user" to "root.users.firstname.lastname", which is likely not what 
we want. It will not work in percentage mode, because "firstname" is a parent 
and you can't create parents under a ManagedParentQueue. In the new weight 
mode, it can work, but again, the intention is to have something like 
"root.users.firstname_lastname", just a single leaf.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291612#comment-17291612
 ] 

Peter Bacsko commented on YARN-10652:
-

[~sahuja] you're right in saying that it has no direct relation in the 
placement.

In the first part of my comment, I was just thinking out loud that MAYBE using 
"_" instead of "." in the property is also a solution, but it comes with its 
own problems.

The placement stuff is different, it's something that we haven't considered so 
far. Currently, the new placement engine simply replaces placeholders like 
"root.users.%user" to "root.users.firstname.lastname", which is likely not what 
we want. It will not work in percentage mode, because "firstname" is a parent 
and you can't create parents under a ManagedParentQueue. In the new weight 
mode, it can work, but again, the intention is to have something like 
"root.users.firstname_lastname", just a single leaf.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291595#comment-17291595
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:53 AM:
---

Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with a dot as an underscore 
internally, it should not force the users who are specifying their users 
containing a dot against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead supply "_" because our s/w needs to store a dot as an 
underscore internally. Substituting a dot with an underscore is internal s/w 
implementation, however, we should not prohibit users from supplying usernames 
with a dot (which are valid usernames in AD) against 
yarn.scheduler.capacity..user-settings..weight. I don't 
believe this is done so in Fair Scheduler either and underscores are internal 
implementation. Again, please correct me if I am wrong [~pbacsko],[~shuzirra].


was (Author: sahuja):
Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with a dot as an underscore 
internally, it should not force the users who are specifying their users 
containing a dot against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead supply "_" because our s/w needs to store a dot as an 
underscore internally. Substituting a dot with an underscore is internal s/w 
implementation, however, we should not prohibit users from supplying usernames 
with a dot (which are valid usernames in AD) against 
yarn.scheduler.capacity..user-settings..weight. I don't 
believe this is done so in Fair Scheduler either and underscores are internal 
implementation. Again, please correct me if I am wrong.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.

[Hadoop QA (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291601#comment-17291601
 ] 

Hadoop QA commented on YARN-10623:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
1s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
36s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
49s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
19s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
39s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m  7s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
57s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
59s{color} | 
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/688/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html{color}
 | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 
1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for 
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
25s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
11s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
15s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
15s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespa

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291595#comment-17291595
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:49 AM:
---

Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with a dot as an underscore 
internally, it should not force the users who are specifying their users 
containing a dot against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead supply "_" because our s/w needs to store a dot as an 
underscore internally. Substituting a dot with an underscore is internal s/w 
implementation, however, we should not prohibit users from supplying usernames 
with a dot (which are valid usernames in AD) against 
yarn.scheduler.capacity..user-settings..weight. I don't 
believe this is done so in Fair Scheduler either and underscores are internal 
implementation. Again, please correct me if I am wrong.


was (Author: sahuja):
Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with a dot as an underscore 
internally, it should not force the users who are specifying their users 
containing a "." against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead provide "_" because our s/w wants to store "_" instead of a 
".". Substituting a "." with an "_" is an internal s/w implementation thing, 
however, we cannot prohibit users from supplying usernames with a "." against 
yarn.scheduler.capacity..user-settings..weight.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291595#comment-17291595
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:47 AM:
---

Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with a dot as an underscore 
internally, it should not force the users who are specifying their users 
containing a "." against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead provide "_" because our s/w wants to store "_" instead of a 
".". Substituting a "." with an "_" is an internal s/w implementation thing, 
however, we cannot prohibit users from supplying usernames with a "." against 
yarn.scheduler.capacity..user-settings..weight.


was (Author: sahuja):
Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with "." as "_" internally, 
it should not force the users who are specifying their users containing a "." 
against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead use "_" because our s/w wants to store "_" instead of a ".". 
Substituting a "." with an "_" is an internal s/w implementation thing, 
however, we cannot prohibit users from supplying usernames with a "." against 
yarn.scheduler.capacity..user-settings..weight.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291595#comment-17291595
 ] 

Siddharth Ahuja commented on YARN-10652:


Also, yarn.scheduler.capacity..user-settings..weight 
setting is a user-specific setting, not an internal implementation. That is to 
say that sure, even if you want to manage usernames with "." as "_" internally, 
it should not force the users who are specifying their users containing a "." 
against this setting - 
"yarn.scheduler.capacity..user-settings..weight setting" 
should now instead use "_" because our s/w wants to store "_" instead of a ".". 
Substituting a "." with an "_" is an internal s/w implementation thing, 
however, we cannot prohibit users from supplying usernames with a "." against 
yarn.scheduler.capacity..user-settings..weight.

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291583#comment-17291583
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:27 AM:
---

Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, the 
yarn.scheduler.capacity..user-settings..weight setting 
has nothing to do with queue placement. We are not trying to have the user - 
firstname.lastname placed into its own queue like root.firstname.lastname here. 
What we are instead trying to solve is that if a user-> firstname.lastname 
wants to submit a job to the root.default queue as an example, then, that user 
can certainly do that today (kindly see the attached screenshots - there is 
nothing stopping that today) and considering it can do that, the user weight 
setting that decides the user's share in the root.default (not 
root.firstname.lastname) queue should be able to cater for this user, that's 
all. Giving firstname.lastname user it's share through this setting does not 
trigger any placements and as such should not have any cause for concern. 
Again, please correct me if I am wrong [~pbacsko].


was (Author: sahuja):
Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, the 
yarn.scheduler.capacity..user-settings..weight setting 
has nothing to do with queue placement. We are not trying to have the user - 
firstname.lastname placed into its own queue like root.firstname.lastname here. 
What we are instead trying to solve is that if a user-> firstname.lastname 
wants to submit a job to the root.default queue as an example, then, that user 
can certainly do that today (kindly see the attached screenshots - there is 
nothing stopping that today) and considering it can do that, the user weight 
setting that decides the user's share in the root.default (not 
root.firstname.lastname) queue should be able to cater for this user, that's 
all. Giving firstname.lastname user it's share through this setting does not 
trigger any placements and as such should not have any cause of concern. Again, 
please correct me if I am wrong [~pbacsko].

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291583#comment-17291583
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:25 AM:
---

Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, the 
yarn.scheduler.capacity..user-settings..weight setting 
has nothing to do with queue placement. We are not trying to have the user - 
firstname.lastname placed into its own queue like root.firstname.lastname here. 
What we are instead trying to solve is that if a user-> firstname.lastname 
wants to submit a job to the root.default queue as an example, then, that user 
can certainly do that today (kindly see the attached screenshots - there is 
nothing stopping that today) and considering it can do that, the user weight 
setting that decides the user's share in the root.default (not 
root.firstname.lastname) queue should be able to cater for this user, that's 
all. Giving firstname.lastname user it's share through this setting does not 
trigger any placements and as such should not have any cause of concern. Again, 
please correct me if I am wrong [~pbacsko].


was (Author: sahuja):
Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, the 
arn.scheduler.capacity..user-settings..weight setting 
has nothing to do with queue placement. We are not trying to have the user -> 
firstname.lastname placed into its own queue like root.firstname.lastname here. 
What we are instead trying to solve is that if a user-> firstname.lastname 
wants to submit a job to the root.default queue as an example, then, that user 
can certainly do that today (kindly see the attached screenshots - there is 
nothing stopping that today) and considering it can do that, the user weight 
setting that decides the user's share in the root.default (not 
root.firstname.lastname) queue should be able to cater for this user, that's 
all. Giving firstname.lastname user it's share through this setting does not 
trigger any placements and as such should not have any cause of concern. Again, 
please correct me if I am wrong [~pbacsko].

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291583#comment-17291583
 ] 

Siddharth Ahuja edited comment on YARN-10652 at 2/26/21, 11:24 AM:
---

Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, the 
arn.scheduler.capacity..user-settings..weight setting 
has nothing to do with queue placement. We are not trying to have the user -> 
firstname.lastname placed into its own queue like root.firstname.lastname here. 
What we are instead trying to solve is that if a user-> firstname.lastname 
wants to submit a job to the root.default queue as an example, then, that user 
can certainly do that today (kindly see the attached screenshots - there is 
nothing stopping that today) and considering it can do that, the user weight 
setting that decides the user's share in the root.default (not 
root.firstname.lastname) queue should be able to cater for this user, that's 
all. Giving firstname.lastname user it's share through this setting does not 
trigger any placements and as such should not have any cause of concern. Again, 
please correct me if I am wrong [~pbacsko].


was (Author: sahuja):
Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, 
{{yarn.scheduler.capacity..user-settings..weight}} 
setting has nothing to do with queue placement. We are not trying to have the 
user -> firstname.lastname placed into its own queue like 
root.firstname.lastname here. What we are instead trying to solve is that if a 
user-> firstname.lastname wants to submit a job to the root.default queue as an 
example, then, that user can certainly do that today (kindly see the attached 
screenshots - there is nothing stopping that today) and considering it can do 
that, the user weight setting that decides the user's share in the root.default 
(not root.firstname.lastname) queue should be able to cater for this user, 
that's all. Giving firstname.lastname user it's share through this setting does 
not trigger any placements and as such should not have any cause of concern. 
Again, please correct me if I am wrong [~pbacsko].

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Siddharth Ahuja (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291583#comment-17291583
 ] 

Siddharth Ahuja commented on YARN-10652:


Thanks [~rreti] & [~pbacsko] for your comments. However, please correct me if I 
am wrong (and I must admit that I am primarily familiar with FairScheduler as 
well), however, 
{{yarn.scheduler.capacity..user-settings..weight}} 
setting has nothing to do with queue placement. We are not trying to have the 
user -> firstname.lastname placed into its own queue like 
root.firstname.lastname here. What we are instead trying to solve is that if a 
user-> firstname.lastname wants to submit a job to the root.default queue as an 
example, then, that user can certainly do that today (kindly see the attached 
screenshots - there is nothing stopping that today) and considering it can do 
that, the user weight setting that decides the user's share in the root.default 
(not root.firstname.lastname) queue should be able to cater for this user, 
that's all. Giving firstname.lastname user it's share through this setting does 
not trigger any placements and as such should not have any cause of concern. 
Again, please correct me if I am wrong [~pbacsko].

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10654) Dots '.' in CSMappingRule path variables should be replaced

[Gergely Pollak (Jira)]

Gergely Pollak created YARN-10654:
-

 Summary: Dots '.' in CSMappingRule path variables should be 
replaced
 Key: YARN-10654
 URL: https://issues.apache.org/jira/browse/YARN-10654
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Gergely Pollak


Dots are used as separators, so we should escape them somehow in the variables 
when substituting them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Peter Bacsko (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291572#comment-17291572
 ] 

Peter Bacsko commented on YARN-10652:
-

"Dot" in the username is clearly a problem. In FS, there is approach in certain 
situations when dots are replaced to underscores ("_").

Quoting the upstream docs:

{noformat}
user: the app is placed into a queue with the name of the user who submitted 
it. Periods in the username will be replace with “_dot_”, i.e. the queue name 
for user “first.last” is “first_dot_last”.

primaryGroup: the app is placed into a queue with the name of the primary group 
of the user who submitted it. Periods in the group name will be replaced with 
“_dot_”, i.e. the queue name for group “one.two” is “one_dot_two”.
{noformat}

Obviously this is slightly different here, because in this case, you'd refer to 
the username as "firstname_lastname" in a static configuration, which could be 
confusing. Also, "firstname.lastname" and "firstname_lastname" would clash 
(unrealistic, but can happen in theory).

But in the placement engine, we should definitely consider what FS does and 
replace "." with "_".

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10652) Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it

[Rudolf Reti (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291563#comment-17291563
 ] 

Rudolf Reti commented on YARN-10652:


[~pbacsko] & [~shuzirra] Can you please check this from a Placement point of 
view? Won't the DOT in the username break that as DOT is used as a delimiter 
there?

> Capacity Scheduler fails to handle user weights for a user that has a "." 
> (dot) in it
> -
>
> Key: YARN-10652
> URL: https://issues.apache.org/jira/browse/YARN-10652
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Attachments: Correct user weight of 0.76 picked up for the user with 
> a dot after the patch.png, Incorrect default user weight of 1.0 being picked 
> for the user with a dot before the patch.png, YARN-10652.001.patch
>
>
> AD usernames can have a "." (dot) in them i.e. they can be of the format -> 
> {{firstname.lastname}}. However, if you specify a username with this format 
> against the Capacity Scheduler setting -> 
> {{yarn.scheduler.capacity.root.default.user-settings.firstname.lastname.weight}},
>  it fails to be applied and is instead assigned the default of 1.0f weight. 
> This renders the user weight feature (being used as a means of setting user 
> priorities for a queue) unusable for such users.
> This limitation comes from [1]. From [1], only word characters (A word 
> character: [a-zA-Z_0-9]) (see [2]) are permissible at the moment which is no 
> good for AD names that contain a "." (dot).
> Similar discussion has been had in a few HADOOP jiras e.g. HADOOP-7050 and 
> HADOOP-15395 and the outcome was to use non-whitespace characters i.e. 
> instead of {{\w+}}, use {{\S+}}.
> We could go down similar path and unblock this feature for the AD usernames 
> with a "." (dot) in them.
> [1] 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1953
> [2] 
> https://docs.oracle.com/javase/tutorial/essential/regex/pre_char_classes.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

[Qi Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291515#comment-17291515
 ] 

Qi Zhu commented on YARN-10532:
---

[~pbacsko] [~gandras]

The test is not related, if you have any other advice?

Thanks.

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> YARN-10532.021.patch, YARN-10532.022.patch, YARN-10532.023.patch, 
> YARN-10532.024.patch, image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

[Hadoop QA (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291497#comment-17291497
 ] 

Hadoop QA commented on YARN-10532:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
13s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 3 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 41s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
46s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green}{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 278 unchanged - 1 fixed = 278 total (was 279) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 55s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-

[jira] [Commented] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.

[Qi Zhu (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291457#comment-17291457
 ] 

Qi Zhu commented on YARN-10623:
---

Fixed the checkstyle in latest patch.

> Capacity scheduler should support refresh queue automatically by a thread 
> policy.
> -
>
> Key: YARN-10623
> URL: https://issues.apache.org/jira/browse/YARN-10623
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10623.001.patch, YARN-10623.002.patch, 
> YARN-10623.003.patch, YARN-10623.004.patch, YARN-10623.005.patch
>
>
> In fair scheduler, it is supported that refresh queue related conf 
> automatically by a thread to reload, but in capacity scheduler we only 
> support to refresh queue related changes by refreshQueues, it is needed for 
> our cluster to realize queue manage.
> cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras]  [~bteke] [~shuzirra]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.

[Qi Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qi Zhu updated YARN-10623:
--
Attachment: YARN-10623.005.patch

> Capacity scheduler should support refresh queue automatically by a thread 
> policy.
> -
>
> Key: YARN-10623
> URL: https://issues.apache.org/jira/browse/YARN-10623
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10623.001.patch, YARN-10623.002.patch, 
> YARN-10623.003.patch, YARN-10623.004.patch, YARN-10623.005.patch
>
>
> In fair scheduler, it is supported that refresh queue related conf 
> automatically by a thread to reload, but in capacity scheduler we only 
> support to refresh queue related changes by refreshQueues, it is needed for 
> our cluster to realize queue manage.
> cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras]  [~bteke] [~shuzirra]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org