[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-21 Thread Wangda Tan (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269479#comment-17269479
 ] 

Wangda Tan commented on YARN-10587:
---

+1, thanks [~zhuqi]

> Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
> -
>
> Key: YARN-10587
> URL: https://issues.apache.org/jira/browse/YARN-10587
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10587.001.patch, YARN-10587.002.patch
>
>
> When introduced YARN-10504.
> The logic related to auto created leaf queue changed.
> The test in testAutoCreateLeafQueueCreation failed, we should fix the Error.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-20 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269036#comment-17269036
 ] 

Hadoop QA commented on YARN-10587:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 2 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 21s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m 22s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green}{color} | {color:green} the

[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-20 Thread zhuqi (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268983#comment-17268983
 ] 

zhuqi commented on YARN-10587:
--

[~wangda] [~sunilg]

Fix related TestAbsoluteResourceConfiguration in latest patch.

 

 

> Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
> -
>
> Key: YARN-10587
> URL: https://issues.apache.org/jira/browse/YARN-10587
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10587.001.patch, YARN-10587.002.patch
>
>
> When introduced YARN-10504.
> The logic related to auto created leaf queue changed.
> The test in testAutoCreateLeafQueueCreation failed, we should fix the Error.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-20 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268821#comment-17268821
 ] 

Hadoop QA commented on YARN-10587:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m 
43s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} No case conflicting files 
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green}{color} | {color:green} The patch does not contain any 
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green}  0m  0s{color} 
| {color:green}test4tests{color} | {color:green} The patch appears to include 1 
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m  8s{color} | {color:green}{color} | {color:green} branch has no errors when 
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private 
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m 
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs 
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK 
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 31s{color} | 
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/523/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
 | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace 
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 35s{color} | {color:green}{color} | {color:green} patch has no errors when 
building and testing our client artifacts. {color} |
|

[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-20 Thread Wangda Tan (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268743#comment-17268743
 ] 

Wangda Tan commented on YARN-10587:
---

Thanks [~zhuqi],

I think the fix is correct, but I want [~sunilg] also take a look at the patch.

> Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
> -
>
> Key: YARN-10587
> URL: https://issues.apache.org/jira/browse/YARN-10587
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10587.001.patch
>
>
> When introduced YARN-10504.
> The logic related to auto created leaf queue changed.
> The test in testAutoCreateLeafQueueCreation failed, we should fix the Error.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.

2021-01-20 Thread zhuqi (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268670#comment-17268670
 ] 

zhuqi commented on YARN-10587:
--

cc [~leftnoteasy] [~bteke]

When i deep into the testAutoCreateLeafQueueCreation failed about :

 
{code:java}
// TODO: Wangda: I think this test case is not correct, Sunil could help look
// into details.
{code}
I find the root reason is , when absolute mode enabled in auto created leaf 
queue, we finally will update cap related for the UI in :
{code:java}
private void deriveCapacityFromAbsoluteConfigurations(String label,
Resource clusterResource, ResourceCalculator rc) {

  /*
   * In case when queues are configured with absolute resources, it is better
   * to update capacity/max-capacity etc w.r.t absolute resource as well. In
   * case of computation, these values wont be used any more. However for
   * metrics and UI, its better these values are pre-computed here itself.
   */

  // 1. Update capacity as a float based on parent's minResource
  float f = rc.divide(clusterResource,
  queueResourceQuotas.getEffectiveMinResource(label),
  parent.getQueueResourceQuotas().getEffectiveMinResource(label));
  queueCapacities.setCapacity(label, Float.isInfinite(f) ? 0 : f);

  // 2. Update max-capacity as a float based on parent's maxResource
  f = rc.divide(clusterResource,
  queueResourceQuotas.getEffectiveMaxResource(label),
  parent.getQueueResourceQuotas().getEffectiveMaxResource(label));
  queueCapacities.setMaximumCapacity(label, Float.isInfinite(f) ? 0 : f);

  // 3. Update absolute capacity as a float based on parent's minResource and
  // cluster resource.
  queueCapacities.setAbsoluteCapacity(label,
  queueCapacities.getCapacity(label) * parent.getQueueCapacities()
  .getAbsoluteCapacity(label));

  // 4. Update absolute max-capacity as a float based on parent's maxResource
  // and cluster resource.
  queueCapacities.setAbsoluteMaximumCapacity(label,
  queueCapacities.getMaximumCapacity(label) * parent.getQueueCapacities()
  .getAbsoluteMaximumCapacity(label));

  // Re-visit max applications for a queue based on absolute capacity if
  // needed.
  if (this instanceof LeafQueue) {
LeafQueue leafQueue = (LeafQueue) this;
CapacitySchedulerConfiguration conf = csContext.getConfiguration();
int maxApplications = conf.getMaximumApplicationsPerQueue(queuePath);
if (maxApplications < 0) {
  int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
  if (maxGlobalPerQueueApps > 0) {
maxApplications = (int) (maxGlobalPerQueueApps * queueCapacities
.getAbsoluteCapacity(label));
  } else{
maxApplications =
(int) (conf.getMaximumSystemApplications() * queueCapacities
.getAbsoluteCapacity(label));
  }
}
leafQueue.setMaxApplications(maxApplications);

int maxApplicationsPerUser = Math.min(maxApplications,
(int) (maxApplications * (leafQueue.getUsersManager().getUserLimit()
/ 100.0f) * leafQueue.getUsersManager().getUserLimitFactor()));
leafQueue.setMaxApplicationsPerUser(maxApplicationsPerUser);
LOG.info("LeafQueue:" + leafQueue.getQueuePath() + ", maxApplications="
+ maxApplications + ", maxApplicationsPerUser="
+ maxApplicationsPerUser + ", Abs Cap:" + queueCapacities
.getAbsoluteCapacity(label) + ", Cap: " + queueCapacities
.getCapacity(label) + ", MaxCap : " + queueCapacities
.getMaximumCapacity(label));
  }
}
{code}
But the queueResourceQuotas.getEffectiveMinResource is not correct when we add 
a absolute auto created leaf queue, just exceed the capacity:
{code:java}
@Override
public AutoCreatedLeafQueueConfig getInitialLeafQueueConfiguration(
AbstractAutoCreatedLeafQueue leafQueue)
throws SchedulerDynamicEditException {

  ...


  float availableCapacity = managedParentQueue.getQueueCapacities().
  getAbsoluteCapacity(nodeLabel) - parentQueueState.
  getAbsoluteActivatedChildQueueCapacity(nodeLabel) + EPSILON;

  if (availableCapacity >= leafQueueTemplateCapacities
  .getAbsoluteCapacity(nodeLabel)) {
   ...
  } else{
updateToZeroCapacity(capacities, nodeLabel, leafQueue);
  }
  ...


}
{code}
In updateToZeroCapacity we should change to ,For absolute auto created leaf 
queue:
{code:java}
private void updateToZeroCapacity(QueueCapacities capacities,
String nodeLabel, LeafQueue leafQueue) {
  capacities.setCapacity(nodeLabel, 0.0f);
  capacities.setMaximumCapacity(nodeLabel,
  leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
  leafQueue.getQueueResourceQuotas().
  setConfiguredMinResource(nodeLabel, Resource.newInstance(0,0));
}
{code}
Then in calculateEffectiveResourcesAndCapacity the ratio will be correct, when 
the absolute auto created leaf queue will not add to. Then the