[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[ https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367477#comment-14367477 ] Sunil G commented on YARN-2693: --- Thank you [~wangda] for sharing comments. As we move in the queue specific config inside scheduler.Queue, are we also taking ACLs back to scheduler (ACL wrt priority). Its better to control ACL from outside YarnAuthorizer and config only can be kept w.r.t scheduler. Pls share your thoughts. Regarding methods in ApplicationPrioirtyManager, it looks overall fine but I suggest we may need * getClusterApplicationPriorities (if its range, that can be sent back) > Priority Label Manager in RM to manage application priority based on > configuration > -- > > Key: YARN-2693 > URL: https://issues.apache.org/jira/browse/YARN-2693 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch, > 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch, > 0006-YARN-2693.patch > > > Focus of this JIRA is to have a centralized service to handle priority labels. > Support operations such as > * Add/Delete priority label to a specified queue > * Manage integer mapping associated with each priority label > * Support managing default priority label of a given queue > * Expose interface to RM to validate priority label > TO have simplified interface, Priority Manager will support only > configuration file in contrast with admin cli and REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[ https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366225#comment-14366225 ] Wangda Tan commented on YARN-2693: -- Some overall suggestions: 1) Instead of ApplicationPriorityPerQueue, queue's priority related fields could store in scheduler.Queue directly (methods of scheduler.Queue, implementation for different scheduler could be various. Since we have different scheduler configuration). Benefits of doing this: - All other queue-specifc configurations are in schedulers' own configuration files, make application-priority fields for queue storing out-of queue means you have to sync it with queue's configuration when you do refreshQueues, etc. - Put it in scheduler.Queue can make scheduler changes easier (don't have to access ApplicationPriorityManager). 2) Methods of ApplicationPriorityManager: - Since we're discussing how to configure priority, I will review ApplicationPriorityManager implementations once we close design. - "ClusterPriorities" should be a range (If we start from zero, a "maxPriority" will be enough) - getApplicationPriorityFromQueue should not exist, all queue related methods should be in scheduler.Queue - "isPriorityExistsInCluster" maybe not "existed", it should be something like "accepted". - Can be reinitialize - Can convert between number/label > Priority Label Manager in RM to manage application priority based on > configuration > -- > > Key: YARN-2693 > URL: https://issues.apache.org/jira/browse/YARN-2693 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch, > 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch, > 0006-YARN-2693.patch > > > Focus of this JIRA is to have a centralized service to handle priority labels. > Support operations such as > * Add/Delete priority label to a specified queue > * Manage integer mapping associated with each priority label > * Support managing default priority label of a given queue > * Expose interface to RM to validate priority label > TO have simplified interface, Priority Manager will support only > configuration file in contrast with admin cli and REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347172#comment-14347172
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702487/0006-YARN-2693.patch
against trunk revision 3560180.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationpriority.TestApplicationPriorityManager
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6837//console
This message is automatically generated.
> Priority Label Manager in RM to manage application priority based on
> configuration
> --
>
> Key: YARN-2693
> URL: https://issues.apache.org/jira/browse/YARN-2693
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
> 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
> 0006-YARN-2693.patch
>
>
> Focus of this JIRA is to have a centralized service to handle priority labels.
> Support operations such as
> * Add/Delete priority label to a specified queue
> * Manage integer mapping associated with each priority label
> * Support managing default priority label of a given queue
> * Expose interface to RM to validate priority label
> TO have simplified interface, Priority Manager will support only
> configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343435#comment-14343435
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701889/0006-YARN-2693.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1158 javac
compiler warnings (more than the trunk's current 1157 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6802//console
This message is automatically generated.
> Priority Label Manager in RM to manage application priority based on
> configuration
> --
>
> Key: YARN-2693
> URL: https://issues.apache.org/jira/browse/YARN-2693
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
> 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
> 0006-YARN-2693.patch
>
>
> Focus of this JIRA is to have a centralized service to handle priority labels.
> Support operations such as
> * Add/Delete priority label to a specified queue
> * Manage integer mapping associated with each priority label
> * Support managing default priority label of a given queue
> * Expose interface to RM to validate priority label
> TO have simplified interface, Priority Manager will support only
> configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343404#comment-14343404
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701883/0006-YARN-2693.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 209 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
47 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6801//console
This message is automatically generated.
> Priority Label Manager in RM to manage application priority based on
> configuration
> --
>
> Key: YARN-2693
> URL: https://issues.apache.org/jira/browse/YARN-2693
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
> 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
> 0006-YARN-2693.patch
>
>
> Focus of this JIRA is to have a centralized service to handle priority labels.
> Support operations such as
> * Add/Delete priority label to a specified queue
> * Manage integer mapping associated with each priority label
> * Support managing default priority label of a given queue
> * Expose interface to RM to validate priority label
> TO have simplified interface, Priority Manager will support only
> configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340015#comment-14340015
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701313/0006-YARN-2693.patch
against trunk revision 48c7ee7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6777//console
This message is automatically generated.
> Priority Label Manager in RM to manage application priority based on
> configuration
> --
>
> Key: YARN-2693
> URL: https://issues.apache.org/jira/browse/YARN-2693
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
> 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
> 0006-YARN-2693.patch
>
>
> Focus of this JIRA is to have a centralized service to handle priority labels.
> Support operations such as
> * Add/Delete priority label to a specified queue
> * Manage integer mapping associated with each priority label
> * Support managing default priority label of a given queue
> * Expose interface to RM to validate priority label
> TO have simplified interface, Priority Manager will support only
> configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338914#comment-14338914
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701122/0006-YARN-2693.patch
against trunk revision dce8b9c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6758//console
This message is automatically generated.
> Priority Label Manager in RM to manage application priority based on
> configuration
> --
>
> Key: YARN-2693
> URL: https://issues.apache.org/jira/browse/YARN-2693
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
> 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
> 0006-YARN-2693.patch
>
>
> Focus of this JIRA is to have a centralized service to handle priority labels.
> Support operations such as
> * Add/Delete priority label to a specified queue
> * Manage integer mapping associated with each priority label
> * Support managing default priority label of a given queue
> * Expose interface to RM to validate priority label
> TO have simplified interface, Priority Manager will support only
> configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
