[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-08 Thread Hudson (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947120#comment-16947120
 ] 

Hudson commented on YARN-9760:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17504 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/17504/])
YARN-9760. Support configuring application priorities on a workflow (jhung: rev 
5462d2176f7a64b236f06700cd247f946798cb26)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerWorkflowPriorityMapping.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/WorkflowPriorityMappingsManager.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java


> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Fix For: 2.10.0, 3.3.0, 3.2.2, 3.1.4
>
> Attachments: YARN-9760-branch-2.001.patch, 
> YARN-9760-branch-3.1.001.patch, YARN-9760-test.patch, YARN-9760.01.patch, 
> YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, 
> YARN-9760.05.patch, YARN-9760.06.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-08 Thread Varun Saxena (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947034#comment-16947034
 ] 

Varun Saxena commented on YARN-9760:


branch-2 and branch-3.1 patches LGTM.
The test failures are the same ones that have been discussed for the trunk 
patch.

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760-branch-2.001.patch, 
> YARN-9760-branch-3.1.001.patch, YARN-9760-test.patch, YARN-9760.01.patch, 
> YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, 
> YARN-9760.05.patch, YARN-9760.06.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-07 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946445#comment-16946445
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
15s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
15s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
20s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
48s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
56s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
14s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 53s{color} | {color:orange} root: The patch generated 28 new + 498 unchanged 
- 3 fixed = 526 total (was 501) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
57s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
7s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
54s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
5s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m 
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 30s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-07 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946433#comment-16946433
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m  
2s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
54s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
36s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
3s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m  
5s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m  
6s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m  
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 58s{color} | {color:orange} root: The patch generated 28 new + 498 unchanged 
- 3 fixed = 526 total (was 501) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
28s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
8s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
18s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 64m 
42s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 49s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-07 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946431#comment-16946431
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
33s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m  
5s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
22s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
44s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
21m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
52s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
42s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 15s{color} | {color:orange} root: The patch generated 20 new + 535 unchanged 
- 3 fixed = 555 total (was 538) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 43s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 33s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
29s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m 
31s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 49s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}339m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyFromLocal |
|   | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-07 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946414#comment-16946414
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
49s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  2m 
54s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 30s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m  
4s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m  4s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 29s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged 
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 36s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
43s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
47s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 54s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 46s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
44s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}302m 31s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-07 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946278#comment-16946278
 ] 

Jonathan Hung commented on YARN-9760:
-

Attached:
 * 06 patch which changes an import in 
TestCapacitySchedulerWorkflowPriorityMapping class
 * branch-3.1.01 patch which adds appTags functionality to MockRM.java (this 
was added in YARN-8015)
 * branch-2.01 patch which adds appTags functionality to MockRM.java, and adds 
some constants to TestCapacitySchedulerWorkflowPriorityMapping (since the 
inherited CapacitySchedulerTestBase does not exist in branch-2)

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760-branch-2.001.patch, 
> YARN-9760-branch-3.1.001.patch, YARN-9760-test.patch, YARN-9760.01.patch, 
> YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, 
> YARN-9760.05.patch, YARN-9760.06.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-06 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945307#comment-16945307
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 56s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} 
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m 
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 52s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged 
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} hadolint {color} | {color:green}  0m  
2s{color} | {color:green} There were no new hadolint issues. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
18s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 32s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} 
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}153m 19s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
57s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}345m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-06 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945279#comment-16945279
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 2s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m  
3s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
21m 23s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m  
4s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m  4s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 41s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged 
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 28s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
37s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 48s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
38s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}315m 46s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9760 |
| JIRA Patch URL |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-05 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945256#comment-16945256
 ] 

Hadoop QA commented on YARN-9760:
-

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-YARN-Build/24913/console in case of 
problems.


> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760-test.patch, YARN-9760.01.patch, 
> YARN-9760.02.patch, YARN-9760.03.patch, YARN-9760.04.patch, YARN-9760.05.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-05 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945241#comment-16945241
 ] 

Jonathan Hung commented on YARN-9760:
-

Compiler error does not seem related. Filed HADOOP-16636 for this.

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch, 
> YARN-9760.03.patch, YARN-9760.04.patch, YARN-9760.05.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-05 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945232#comment-16945232
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m  
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
34s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  3m 
54s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
22m 59s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  4m 
11s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  4m 11s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
3m 21s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged 
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 36s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
39s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 11s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 10s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
37s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}320m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerOvercommit
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-05 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945158#comment-16945158
 ] 

Jonathan Hung commented on YARN-9760:
-

04 seems fine. Attached 05 which addresses the nit above. Other than that, +1. 
I'll commit this shortly.

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch, 
> YARN-9760.03.patch, YARN-9760.04.patch, YARN-9760.05.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-04 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944915#comment-16944915
 ] 

Jonathan Hung commented on YARN-9760:
-

Actually, happened upon another nit. Can we rename 
getMappedWorkflowPriorityForApp to something like mapWorkflowPriorityForApp? 
The name "getMappedWorkflowPriorityForApp" suggests it's just a getter, but we 
potentially mutate the ASC's priority. 

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch, 
> YARN-9760.03.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-04 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944890#comment-16944890
 ] 

Jonathan Hung commented on YARN-9760:
-

[~varun_saxena], thanks for the 03 patch. I was able to get the TestMRJobs test 
case to succeed by taking out the lines you added in the setup function:
{noformat}
conf.set(YarnConfiguration.SCHEDULER_CONFIGURATION_STORE_CLASS,  
YarnConfiguration.MEMORY_CONFIGURATION_STORE); {noformat}
and taking out the hadoop-mapreduce-client-jobclient capacity-scheduler.xml 
file.  Seems like this might be a bug with the mutable cs configuration 
provider (but unrelated to this patch).

Could you attach a patch with these two changes? Thanks!

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch, 
> YARN-9760.03.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-10-02 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943208#comment-16943208
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
20m 15s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 41s{color} | {color:orange} root: The patch generated 19 new + 534 unchanged 
- 3 fixed = 553 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 21s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m  
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
1s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m  
1s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 
41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 30s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
52s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}338m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9760 |
| JIRA Patch URL |

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-27 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939818#comment-16939818
 ] 

Jonathan Hung commented on YARN-9760:
-

Thanks [~varun_saxena]. Seems fine. A couple nits:
 * Do we need to check in 
{{hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/capacity-scheduler.xml}}
 ?
 * The YarnConfiguration comment is a bit confusing:
{noformat}
// Configuration for the prefix of the tag which contains workflow ID,
// followed by the prefix. {noformat}
Can we reword this? Maybe "Prefix of an application tag which is followed by 
that application's workflow id"
 * There's a stray "{{as the user."}}  line in the description for 
yarn.scheduler.capacity.workflow-priority-mappings (in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml)
 * Can we make CapacitySchedulerConfiguration#getWorkflowPriorityMappings 
return Collection instead? (also getWorkFlowPriorityMappings -> 
getWorkflowPriorityMappings)
 * Can we push {{if (overrideWithPriorityMappings) {}} in 
WorkflowPriorityMappingsManager#getMappedWorkflowPriorityForApp to the top of 
this method? That way there's no impact if this feature's not enabled.
 * import java.util.HashSet; is unused in MockRM.java

Looks good other than that though.

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-26 Thread Varun Saxena (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938635#comment-16938635
 ] 

Varun Saxena commented on YARN-9760:


MAPREDUCE-7231 tracks hadoop-mapreduce-client-jobclient timeout. Related to 
TestPipeApplication. Going by MAPREDUCE-7036, the ASF license error may also be 
related.
TestFairSchedulerPreemption failure is tracked by YARN-9333.

Will fix some of the checkstyle issues post the review.

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch, YARN-9760.02.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-19 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933707#comment-16933707
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 37m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 56s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 25s{color} | {color:orange} root: The patch generated 20 new + 534 unchanged 
- 3 fixed = 554 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 50s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
53s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
52s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 10s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 28s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  1m 
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}379m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-19 Thread Hadoop QA (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933124#comment-16933124
 ] 

Hadoop QA commented on YARN-9760:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 10s{color} 
| {color:red} YARN-9760 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9760 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12980334/YARN-9760.01.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/24806/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-14 Thread Varun Saxena (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929815#comment-16929815
 ] 

Varun Saxena commented on YARN-9760:


Attached a patch as a reference to what we are planning to do

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
> Attachments: YARN-9760.01.patch
>
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-09-14 Thread Varun Saxena (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929810#comment-16929810
 ] 

Varun Saxena commented on YARN-9760:


Sorry, I missed the comments here. [~eepayne], as Jonathan explained we wanted 
cluster admins to control the priority, hence the feature.

[~jhung], we can use YARN tags to get workflow information as we do in timeline 
service. By leveraging queue mapping interface, I guess you mean the 
PlacementRule interface. That though is meant primarily for queue placement, 
the way it is defined right now. We can potentially have a new interface, once 
we have more other use cases for setting priority i.e. based on something other 
than workflows?

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-08-22 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913621#comment-16913621
 ] 

Jonathan Hung commented on YARN-9760:
-

Hi [~eepayne], thanks for the comments. I understand the concern. The reason we 
implemented it this way was b/c we wanted queue admins to be able to set 
workflow priorities themselves (i.e. leveraging YARN queue ACLs) which would be 
difficult to do client-side.

IMO there's not too much complexity for this feature on the RM side. But to 
minimize overall complexity, perhaps we can do this in a way that doesn't make 
protocol changes, and on the RM side we can leverage the queue mapping 
interface. Would this address your concerns? [~varun_saxena] how does this 
sound?

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9760) Support configuring application priorities on a workflow level

2019-08-21 Thread Eric Payne (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912521#comment-16912521
 ] 

Eric Payne commented on YARN-9760:
--

[~jhung], thanks for raising this interesting feature request.

In order to avoid making the RM even more complex than it already is, I am a 
little concerned about adding more features to YARN that could possibly be done 
different ways. For example, creating separate subqueues for different 
priorities and using the queue priority feature. Or, writing a custom script on 
the client side that that does this analysis at submission time.

I just want to avoid further complicating YARN if we don't have to. Thoughts?

> Support configuring application priorities on a workflow level
> --
>
> Key: YARN-9760
> URL: https://issues.apache.org/jira/browse/YARN-9760
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Jonathan Hung
>Assignee: Varun Saxena
>Priority: Major
>  Labels: release-blocker
>
> Currently priorities are submitted on an application level, but for end users 
> it's common to submit workloads to YARN at a workflow level. This jira 
> proposes a feature to store workflow id + priority mappings on RM (similar to 
> queue mappings). If app is submitted with a certain workflow id (as set in 
> application submission context) RM will override this app's priority with the 
> one defined in the mapping.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org