[jira] [Created] (YARN-1121) RMStateStore should flush all pending store events before closing
Bikas Saha created YARN-1121: Summary: RMStateStore should flush all pending store events before closing Key: YARN-1121 URL: https://issues.apache.org/jira/browse/YARN-1121 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Bikas Saha Fix For: 2.1.1-beta on serviceStop it should wait for all internal pending events to drain before stopping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out stateless services from stateful services in the RM
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753368#comment-13753368 ] Bikas Saha commented on YARN-1098: -- Overall approach looks good. We probably need a distinction between Active services and AlwaysOn services. Active services are started on transitionToActive() and stopped on transitionToStandby(). eg. amliveliness monitor should stop in standby mode. Without digging into the details, IMO, only the RPC services and perhaps the dispatcher should be in AlwaysOn services. Distinction between stateless and stateful services is probably not the useful in this context. YARN-1027 also introduced a similar concept of standby services and active services. AlwaysOn service run in both active and standby. The addIfService refactoring, if needed, would probably be a different jira. Dont see why we would require it for this change. ResourceManager can have an internal composite service called ActiveStateServices that is used to start and stop active services. AlwaysOn services could start and stop within the ResourceManager composite service as they currently do. They probably dont need to be differentiated. Other than services, there are objects like RMAppManager, RMStateStore, RMContext (that has all apps etc), ResourceScheduler, RMNMInfo etc. that need to be started during transitionToActive() and stopped during transitionToStandby(). We could choose to either 1) discard these objects and hope they get GC'd or 2) explicitly add a start/stop method to them in which they can discard their internal state and become clean to refresh their new state from the state store at a later point. For some of them, it may be easier to discard the object and for others it may be easier to call a stop() method. Separate out stateless services from stateful services in the RM Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[
https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753378#comment-13753378
]
Bikas Saha commented on YARN-707:
-
please feel free to define this where it makes most sense. The existing
AM_RM_TOKEN_SERVICE is a stop gap to be fixed via YARN-986. If you want, then
please move it also.
{code}
AM_CLIENT_TOKEN_MASTER_KEY_NAME
{code}
Is there a way for us to check that the clientAMSecretManager also got
refreshed with the same key?
{code}
+// assert client token master key is recovered back to api-versioned
+// client token master key
+Assert.assertEquals(client token master key not restored,
+attempt1.getClientTokenMasterKey(),
+loadedAttempt1.getClientTokenMasterKey());
{code}
For the rest I will defer to comments from previous reviewers.
Add user info in the YARN ClientToken
-
Key: YARN-707
URL: https://issues.apache.org/jira/browse/YARN-707
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Jason Lowe
Priority: Blocker
Fix For: 3.0.0, 2.1.1-beta
Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt,
YARN-707-20130828-2.txt, YARN-707-20130828.txt
If user info is present in the client token then it can be used to do limited
authz in the AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[
https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753390#comment-13753390
]
Bikas Saha commented on YARN-771:
-
What if we add node to blacklist and them remove that node from blacklist
immediately? Wont it end up in both list?
Also, should it be an error to specify the same node in both lists in the same
method call? In that too, we may end up with the same node in both lists.
{code}
+ @Override
+ public synchronized void updateBlackList(ListString blacklistAdditions,
+ ListString blacklistRemovals) {
+if (blacklistAdditions != null) {
+ this.blacklistAdditions.addAll(blacklistAdditions);
+ // if some nodes are also in blacklistRemovals updated before, we should
+ // remove them here.
+ this.blacklistRemovals.removeAll(blacklistAdditions);
+}
+
+if (blacklistRemovals != null) {
+ // we don't need to handle blacklistAdditions as if same nodes are
duplicated
+ // in blacklistAdditions, only blacklistRemovals works and nodes will be
+ // removed from black list.
+ this.blacklistRemovals.addAll(blacklistRemovals);
+}
+ }
{code}
Can we modify TestAMRMClient.testAllocation() to check that on an exception
during allocate we continue to preserve the previous blacklist and any new
blacklist additions that were made during the allocate call. The test already
has support for this. We just need to augment it with blacklist operations.
Minor nit Blacklist reads better than BlackList.
AMRMClient support for resource blacklisting
-
Key: YARN-771
URL: https://issues.apache.org/jira/browse/YARN-771
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Junping Du
Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, YARN-771-v3.patch
After YARN-750 AMRMClient should support blacklisting via the new YARN API's
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753477#comment-13753477 ] Junping Du commented on YARN-771: - [~bikassaha] b.q. What if we add node to blacklist and them remove that node from blacklist immediately? Wont it end up in both list? Yes. it will appear in both list. However, when going to RM, it will add node first and remove it from blacklist set. So node will not be blacklisting. b.q. Also, should it be an error to specify the same node in both lists in the same method call? In that too, we may end up with the same node in both lists. I may have different idea on this. In general, it is not recommended to put the same node on both lists. However, it may not belonging a wrong behaviour. Let's say in some cases, application want to put nodes that meet requirement A (set A) into blacklist, but don't want to put nodes in requirement B (set B) into list. Application can easily to put set A on additional list and set B on removal list in the same call without taking care the overlap of A and B. Thoughts? b.q. Can we modify TestAMRMClient.testAllocation() to check that on an exception during allocate we continue to preserve the previous blacklist and any new blacklist additions that were made during the allocate call. The test already has support for this. We just need to augment it with blacklist operations. Sure. Thanks for suggestions here and last readable comments. AMRMClient support for resource blacklisting - Key: YARN-771 URL: https://issues.apache.org/jira/browse/YARN-771 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Junping Du Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, YARN-771-v3.patch After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1034) Remove experimental in the Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753503#comment-13753503 ] Hudson commented on YARN-1034: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #316 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/316/]) YARN-1034. Remove experimental in the Fair Scheduler documentation. (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518444) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Remove experimental in the Fair Scheduler documentation - Key: YARN-1034 URL: https://issues.apache.org/jira/browse/YARN-1034 Project: Hadoop YARN Issue Type: Task Components: documentation, scheduler Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Karthik Kambatla Priority: Trivial Labels: doc Fix For: 2.1.1-beta Attachments: yarn-1034-1.patch The YARN Fair Scheduler is largely stable now, and should no longer be declared experimental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1101) Active nodes can be decremented below 0
[ https://issues.apache.org/jira/browse/YARN-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753502#comment-13753502 ] Hudson commented on YARN-1101: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #316 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/316/]) YARN-1101. Active nodes can be decremented below 0 (Robert Parker via tgraves_ (tgraves: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518384) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java Active nodes can be decremented below 0 --- Key: YARN-1101 URL: https://issues.apache.org/jira/browse/YARN-1101 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.0.6-alpha Reporter: Robert Parker Assignee: Robert Parker Fix For: 3.0.0, 2.3.0, 0.23.10, 2.1.1-beta Attachments: YARN-1101_b0.23_v1.patch, YARN-1101_v1.patch The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1101) Active nodes can be decremented below 0
[ https://issues.apache.org/jira/browse/YARN-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753520#comment-13753520 ] Hudson commented on YARN-1101: -- FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #714 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/714/]) YARN-1101. Active nodes can be decremented below 0 (Robert Parker via tgraves) (tgraves: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518385) * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java Active nodes can be decremented below 0 --- Key: YARN-1101 URL: https://issues.apache.org/jira/browse/YARN-1101 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.0.6-alpha Reporter: Robert Parker Assignee: Robert Parker Fix For: 3.0.0, 2.3.0, 0.23.10, 2.1.1-beta Attachments: YARN-1101_b0.23_v1.patch, YARN-1101_v1.patch The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: YARN-696.diff Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: (was: YARN-696.diff) Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1034) Remove experimental in the Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753613#comment-13753613 ] Hudson commented on YARN-1034: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1506 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1506/]) YARN-1034. Remove experimental in the Fair Scheduler documentation. (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518444) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Remove experimental in the Fair Scheduler documentation - Key: YARN-1034 URL: https://issues.apache.org/jira/browse/YARN-1034 Project: Hadoop YARN Issue Type: Task Components: documentation, scheduler Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Karthik Kambatla Priority: Trivial Labels: doc Fix For: 2.1.1-beta Attachments: yarn-1034-1.patch The YARN Fair Scheduler is largely stable now, and should no longer be declared experimental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1034) Remove experimental in the Fair Scheduler documentation
[ https://issues.apache.org/jira/browse/YARN-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753673#comment-13753673 ] Hudson commented on YARN-1034: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1533 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1533/]) YARN-1034. Remove experimental in the Fair Scheduler documentation. (Karthik Kambatla via Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518444) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Remove experimental in the Fair Scheduler documentation - Key: YARN-1034 URL: https://issues.apache.org/jira/browse/YARN-1034 Project: Hadoop YARN Issue Type: Task Components: documentation, scheduler Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Karthik Kambatla Priority: Trivial Labels: doc Fix For: 2.1.1-beta Attachments: yarn-1034-1.patch The YARN Fair Scheduler is largely stable now, and should no longer be declared experimental. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1101) Active nodes can be decremented below 0
[ https://issues.apache.org/jira/browse/YARN-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753672#comment-13753672 ] Hudson commented on YARN-1101: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1533 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1533/]) YARN-1101. Active nodes can be decremented below 0 (Robert Parker via tgraves_ (tgraves: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518384) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java Active nodes can be decremented below 0 --- Key: YARN-1101 URL: https://issues.apache.org/jira/browse/YARN-1101 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.0.6-alpha Reporter: Robert Parker Assignee: Robert Parker Fix For: 3.0.0, 2.3.0, 0.23.10, 2.1.1-beta Attachments: YARN-1101_b0.23_v1.patch, YARN-1101_v1.patch The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: (was: YARN-696.diff) Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: YARN-696.diff Changed waitForState to RUNNING in Junit test Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1119) Add ClusterMetrics checks to tho TestRMNodeTransitions tests
[ https://issues.apache.org/jira/browse/YARN-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated YARN-1119: Assignee: (was: Robert Parker) Add ClusterMetrics checks to tho TestRMNodeTransitions tests Key: YARN-1119 URL: https://issues.apache.org/jira/browse/YARN-1119 Project: Hadoop YARN Issue Type: Test Components: resourcemanager Affects Versions: 3.0.0, 0.23.9, 2.0.6-alpha Reporter: Robert Parker YARN-1101 identified an issue where UNHEALTHY nodes could double decrement the active nodes. We should add checks for RUNNING node transitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[
https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753750#comment-13753750
]
Hadoop QA commented on YARN-696:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600594/YARN-696.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1793//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1793//console
This message is automatically generated.
Enable multiple states to to be specified in Resource Manager apps REST call
Key: YARN-696
URL: https://issues.apache.org/jira/browse/YARN-696
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Assignee: Trevor Lorimer
Attachments: YARN-696.diff
Within the YARN Resource Manager REST API the GET call which returns all
Applications can be filtered by a single State query parameter (http://rm
http address:port/ws/v1/cluster/apps).
There are 8 possible states (New, Submitted, Accepted, Running, Finishing,
Finished, Failed, Killed), if no state parameter is specified all states are
returned, however if a sub-set of states is required then multiple REST calls
are required (max. of 7).
The proposal is to be able to specify multiple states in a single REST call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1118) Improve help message for $ yarn node
[ https://issues.apache.org/jira/browse/YARN-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-1118: --- Assignee: Xuan Gong Improve help message for $ yarn node Key: YARN-1118 URL: https://issues.apache.org/jira/browse/YARN-1118 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1117) Improve help message for $ yarn applications
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-1117: --- Assignee: Xuan Gong Improve help message for $ yarn applications Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1122) FairScheduler user-as-default-queue always defaults to 'default'
Lohit Vijayarenu created YARN-1122:
--
Summary: FairScheduler user-as-default-queue always defaults to
'default'
Key: YARN-1122
URL: https://issues.apache.org/jira/browse/YARN-1122
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Lohit Vijayarenu
By default YARN fairscheduler should use user name as queue name, but we see
that in our clusters all jobs were ending up in default queue. Even after
picking YARN-333 which is part of trunk, the behavior remains the same. Jobs do
end up in right queue, but from UI perspective they are shown as running under
default queue. It looks like there is small bug with
{noformat}
RMApp rmApp = rmContext.getRMApps().get(applicationAttemptId);
{noformat}
which should actually be
{noformat}
RMApp rmApp =
rmContext.getRMApps().get(applicationAttemptId.getApplicationId());
{noformat}
There is also a simple js change needed for filtering of jobs on fairscheduler
UI page.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1118) Improve help message for $ yarn node
[ https://issues.apache.org/jira/browse/YARN-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753792#comment-13753792 ] Xuan Gong commented on YARN-1118: - Let us fix YARN-1117 and YARN-1118 together in one patch. Improve help message for $ yarn node Key: YARN-1118 URL: https://issues.apache.org/jira/browse/YARN-1118 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1118) Improve help message for $ yarn node
[ https://issues.apache.org/jira/browse/YARN-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-1118. - Resolution: Fixed Improve help message for $ yarn node Key: YARN-1118 URL: https://issues.apache.org/jira/browse/YARN-1118 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1117) Improve help message for $ yarn applications
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1117: Description: There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node (was: There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications) Improve help message for $ yarn applications Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1117: Summary: Improve help message for $ yarn applications and $yarn node (was: Improve help message for $ yarn applications) Improve help message for $ yarn applications and $yarn node --- Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1122) FairScheduler user-as-default-queue always defaults to 'default'
[
https://issues.apache.org/jira/browse/YARN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lohit Vijayarenu updated YARN-1122:
---
Attachment: YARN-1122.1.patch
Simple patch to fix this.
FairScheduler user-as-default-queue always defaults to 'default'
Key: YARN-1122
URL: https://issues.apache.org/jira/browse/YARN-1122
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Lohit Vijayarenu
Attachments: YARN-1122.1.patch
By default YARN fairscheduler should use user name as queue name, but we see
that in our clusters all jobs were ending up in default queue. Even after
picking YARN-333 which is part of trunk, the behavior remains the same. Jobs
do end up in right queue, but from UI perspective they are shown as running
under default queue. It looks like there is small bug with
{noformat}
RMApp rmApp = rmContext.getRMApps().get(applicationAttemptId);
{noformat}
which should actually be
{noformat}
RMApp rmApp =
rmContext.getRMApps().get(applicationAttemptId.getApplicationId());
{noformat}
There is also a simple js change needed for filtering of jobs on
fairscheduler UI page.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1122) FairScheduler user-as-default-queue always defaults to 'default'
[
https://issues.apache.org/jira/browse/YARN-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753804#comment-13753804
]
Sandy Ryza commented on YARN-1122:
--
Thanks for taking this up, [~lohit]. Would it be possible to add a test?
FairScheduler user-as-default-queue always defaults to 'default'
Key: YARN-1122
URL: https://issues.apache.org/jira/browse/YARN-1122
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.0.5-alpha
Reporter: Lohit Vijayarenu
Attachments: YARN-1122.1.patch
By default YARN fairscheduler should use user name as queue name, but we see
that in our clusters all jobs were ending up in default queue. Even after
picking YARN-333 which is part of trunk, the behavior remains the same. Jobs
do end up in right queue, but from UI perspective they are shown as running
under default queue. It looks like there is small bug with
{noformat}
RMApp rmApp = rmContext.getRMApps().get(applicationAttemptId);
{noformat}
which should actually be
{noformat}
RMApp rmApp =
rmContext.getRMApps().get(applicationAttemptId.getApplicationId());
{noformat}
There is also a simple js change needed for filtering of jobs on
fairscheduler UI page.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753806#comment-13753806 ] Bikas Saha commented on YARN-771: - The RM must handle anything input that the user sends to it. However, as the official client, AMRMClient should only send valid input to the RM. IMO, having same node in both list at the same invocation of AMRMClient.updateBlacklist() should be an error. Something is wrong with the user and results may be unexpected. We should detect it and at least log a warning. Adding and removing of nodes in different invocations of the method should result in the expected behavior == the result of the last invocation. It should not be the case that the node ends up in both lists since then the results are indeterminate (in this case the RM logic ends up not blacklisting it). AMRMClient support for resource blacklisting - Key: YARN-771 URL: https://issues.apache.org/jira/browse/YARN-771 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Junping Du Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, YARN-771-v3.patch After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (YARN-1118) Improve help message for $ yarn node
[ https://issues.apache.org/jira/browse/YARN-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reopened YARN-1118: --- Improve help message for $ yarn node Key: YARN-1118 URL: https://issues.apache.org/jira/browse/YARN-1118 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1118) Improve help message for $ yarn node
[ https://issues.apache.org/jira/browse/YARN-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved YARN-1118. --- Resolution: Duplicate Improve help message for $ yarn node Key: YARN-1118 URL: https://issues.apache.org/jira/browse/YARN-1118 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753829#comment-13753829 ] Zhijie Shen commented on YARN-696: -- [~trevlo78], one suggestion: please do not remove the old patches, such that the community can trace the work log. For example, you can name you patches YARN-696.1.patch, YARN-696.20130829.patch or with your own ordering method. Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: (was: YARN-696.diff) Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Trevor Lorimer updated YARN-696: Attachment: YARN-696.diff Currently the MockRM seems to only produce applications with the state of ACCEPTED. Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1080) Improve help message for $ yarn logs
[
https://issues.apache.org/jira/browse/YARN-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753865#comment-13753865
]
Vinod Kumar Vavilapalli commented on YARN-1080:
---
+1, looks good. Checking this in.
Improve help message for $ yarn logs
Key: YARN-1080
URL: https://issues.apache.org/jira/browse/YARN-1080
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1080.1.patch, YARN-1080.2.patch
There are 2 parts I am proposing in this jira. They can be fixed together in
one patch.
1. Standardize help message for required parameter of $ yarn logs
YARN CLI has a command logs ($ yarn logs). The command always requires a
parameter of -applicationId arg. However, help message of the command
does not make it clear. It lists -applicationId as optional parameter. If I
don't set it, YARN CLI will complain this is missing. It is better to use
standard required notation used in other Linux command for help message. Any
user familiar to the command can understand that this parameter is needed
more easily.
{code:title=current help message}
-bash-4.1$ yarn logs
usage: general options are:
-applicationId arg ApplicationId (required)
-appOwner argAppOwner (assumed to be current user if not
specified)
-containerId arg ContainerId (must be specified if node address is
specified)
-nodeAddress arg NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
{code:title=proposed help message}
-bash-4.1$ yarn logs
usage: yarn logs -applicationId application ID [OPTIONS]
general options are:
-appOwner argAppOwner (assumed to be current user if not
specified)
-containerId arg ContainerId (must be specified if node address is
specified)
-nodeAddress arg NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
2. Add description for help command. As far as I know, a user cannot get logs
for running job. Since I spent some time trying to get logs of running
applications, it should be nice to say this in command description.
{code:title=proposed help}
Retrieve logs for completed/killed YARN application
usage: general options are...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[
https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753872#comment-13753872
]
Hadoop QA commented on YARN-696:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600615/YARN-696.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1794//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1794//console
This message is automatically generated.
Enable multiple states to to be specified in Resource Manager apps REST call
Key: YARN-696
URL: https://issues.apache.org/jira/browse/YARN-696
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Assignee: Trevor Lorimer
Attachments: YARN-696.diff
Within the YARN Resource Manager REST API the GET call which returns all
Applications can be filtered by a single State query parameter (http://rm
http address:port/ws/v1/cluster/apps).
There are 8 possible states (New, Submitted, Accepted, Running, Finishing,
Finished, Failed, Killed), if no state parameter is specified all states are
returned, however if a sub-set of states is required then multiple REST calls
are required (max. of 7).
The proposal is to be able to specify multiple states in a single REST call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1080) Improve help message for $ yarn logs
[
https://issues.apache.org/jira/browse/YARN-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753885#comment-13753885
]
Hudson commented on YARN-1080:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4343 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4343/])
YARN-1080. Improved help message for yarn logs command. Contributed by Xuan
Gong. (vinodkv:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518731)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestLogDumper.java
Improve help message for $ yarn logs
Key: YARN-1080
URL: https://issues.apache.org/jira/browse/YARN-1080
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Fix For: 2.1.1-beta
Attachments: YARN-1080.1.patch, YARN-1080.2.patch
There are 2 parts I am proposing in this jira. They can be fixed together in
one patch.
1. Standardize help message for required parameter of $ yarn logs
YARN CLI has a command logs ($ yarn logs). The command always requires a
parameter of -applicationId arg. However, help message of the command
does not make it clear. It lists -applicationId as optional parameter. If I
don't set it, YARN CLI will complain this is missing. It is better to use
standard required notation used in other Linux command for help message. Any
user familiar to the command can understand that this parameter is needed
more easily.
{code:title=current help message}
-bash-4.1$ yarn logs
usage: general options are:
-applicationId arg ApplicationId (required)
-appOwner argAppOwner (assumed to be current user if not
specified)
-containerId arg ContainerId (must be specified if node address is
specified)
-nodeAddress arg NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
{code:title=proposed help message}
-bash-4.1$ yarn logs
usage: yarn logs -applicationId application ID [OPTIONS]
general options are:
-appOwner argAppOwner (assumed to be current user if not
specified)
-containerId arg ContainerId (must be specified if node address is
specified)
-nodeAddress arg NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
2. Add description for help command. As far as I know, a user cannot get logs
for running job. Since I spent some time trying to get logs of running
applications, it should be nice to say this in command description.
{code:title=proposed help}
Retrieve logs for completed/killed YARN application
usage: general options are...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out stateless services from stateful services in the RM
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753898#comment-13753898 ] Karthik Kambatla commented on YARN-1098: Thanks Bikas. Agree with all your points. While working out the patch, I was not entirely happy calling the Standby services stateless because they do have some state. Will post a patch along the lines you suggest. Separate out stateless services from stateful services in the RM Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1057) Add mechanism to check validity of a Node to be Added/Excluded
[ https://issues.apache.org/jira/browse/YARN-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753992#comment-13753992 ] yeshavora commented on YARN-1057: - Hitesh, the usecase will be when invalid hosts are added in include/exclude file, Yarn should recognize them and does not add/remove node from cluster. Yarn should not print below message. INFO util.HostsFileReader (HostsFileReader.java:readFileToSet(68)) - Adding invalidhost.net to the list of included hosts from /tmp/yarn.include Ideally it should say something like java.net.UnknownHostException: invalidhost.net I believe RM shutdown is not needed as long as It can verify existence of a host. Add mechanism to check validity of a Node to be Added/Excluded -- Key: YARN-1057 URL: https://issues.apache.org/jira/browse/YARN-1057 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Attachments: YARN-1057.1.patch Yarn does not complain while passing an invalid hostname like 'invalidhost.com' inside include/exclude node file. (specified by 'yarn.resourcemanager.nodes.include-path' or 'yarn.resourcemanager.nodes.exclude-path'). Need to add a mechanism to check the validity of the hostname before including or excluding from cluster. It should throw an error / exception while adding/removing an invalid node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754044#comment-13754044 ] Omkar Vinit Joshi commented on YARN-1107: - Thanks [~jianhe] for reviewing.. bq. No need to add this method 'compareRMToken' to compare Token set.The following is from JAVA Doc regarding set.equals(): Returns true if the specified object is also a set, the two sets have the same size, and every member of the specified set is contained in this set (or equivalently, every member of this set is contained in the specified set) Yes you are right but I don't want to compare masterKeyId field of the identifier as it will get renewed and updated. Hence added my custom method. bq. TestSecurityMockRM.serviceInit(): the whole method can be removed. Yes you are right.. removing it. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754059#comment-13754059 ] Jason Lowe commented on YARN-1107: -- bq. Jason Lowe and Daryn Sharp, can you confirm if this patch will work? If the RM port is ephemeral, it will be an issue. If this ends up breaking the ephemeral port case that would be unfortunate, since miniclusters run with them and it's very convenient for that use-case. I recall [~daryn] mentioning that we may need to set the bind address twice, once during serviceInit as this patch does to solve the issue and again in serviceStart once we know the actual bind address to help solve the minicluster/ephemeral port case. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out stateless services from stateful services in the RM
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754070#comment-13754070 ] Karthik Kambatla commented on YARN-1098: Thinking more about it, I think we need to have three kinds of services: Always On, Active, and Standby. To begin with (1) Always On can have the admin services and RMHAProtocolService, (2) Active has all other services, (3) Standby has no services. Eventually, once we are done with all the HA work: (1) we can add RPC services that can always run - behave appropriately in the Active/Standby modes, (2) Standby has services that are to be run in the Standby mode alone e.g. HTTPRedirector for webapp assuming webapp doesn't handle the HA state, (3) Active has services required by the Active RM outside of the Always On services. Separate out stateless services from stateful services in the RM Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1117: Attachment: YARN-1117.1.patch Improve help message for $ yarn applications and $yarn node --- Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: YARN-1117.1.patch There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[
https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754101#comment-13754101
]
Hadoop QA commented on YARN-1117:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600652/YARN-1117.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1795//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1795//console
This message is automatically generated.
Improve help message for $ yarn applications and $yarn node
---
Key: YARN-1117
URL: https://issues.apache.org/jira/browse/YARN-1117
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1117.1.patch
There is standardization of help message in YARN-1080. It is nice to have
similar changes for $ yarn appications and yarn node
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out stateless services from stateful services in the RM
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754102#comment-13754102 ] Bikas Saha commented on YARN-1098: -- Are there services that need to run in Standby mode but not in Active mode? I dont think there are any because currently there is no standby mode and every service runs in the active mode. Hence, a Standby services group is not clear to me since its an empty group. Separate out stateless services from stateful services in the RM Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754109#comment-13754109 ] Vinod Kumar Vavilapalli commented on YARN-1107: --- We can definitely set the bind-address twice, [~ojoshi] also mentioned the same. Even otherwise, we don't have secure tests with MiniYARNClusters where we validate delegation-tokens - we only have unit tests; so we should be fine. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-707: Attachment: YARN-707-20130829.txt Patch to pass the client user name to getApplicationReport and enhance the restart test to verify the secret manager also knows about the key after restart. I left AM_CLIENT_TOKEN_MASTER_KEY_NAME in RMStateStore since there are many reasons why RMAppAttemptImpl is importing RMStateStore, not just for this constant, and the only reason it needs the constant is for storage purposes. Cleaning up the interface between RMAppAttemptImpl and RMStateStore is another JIRA, IMHO. Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Jason Lowe Priority: Blocker Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754140#comment-13754140 ] Vinod Kumar Vavilapalli commented on YARN-1107: --- Thinking back, I am increasingly getting uncomfortable with this solution, it all seems like a hack. How about this solution (which incidentally was the first solution that occurred to me): - Start the TokenRenewer only after ClientRMService starts. - During recovery, just enqueue all the token-requests - It is TokenRenewer's responsibility to set up the short-cut. ClientRMService shouldn't have any knowledge of this. - Globally, there is a dependency of starting TokenRenewer after ClientRMService. This seems much natural to me. Shouldn't be too much of a change too. Thoughts? Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[
https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754149#comment-13754149
]
Hadoop QA commented on YARN-707:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600659/YARN-707-20130829.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1796//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1796//console
This message is automatically generated.
Add user info in the YARN ClientToken
-
Key: YARN-707
URL: https://issues.apache.org/jira/browse/YARN-707
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Jason Lowe
Priority: Blocker
Fix For: 3.0.0, 2.1.1-beta
Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt,
YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt
If user info is present in the client token then it can be used to do limited
authz in the AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1117: Attachment: YARN-1117.2.patch Improve help message for $ yarn applications and $yarn node --- Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: YARN-1117.1.patch, YARN-1117.2.patch There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[ https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-981: - Attachment: YARN-981.4.patch New patch added a unit test YARN/MR2/Job-history /logs link does not have correct content - Key: YARN-981 URL: https://issues.apache.org/jira/browse/YARN-981 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch, YARN-981.4.patch, YARN-981.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-696) Enable multiple states to to be specified in Resource Manager apps REST call
[ https://issues.apache.org/jira/browse/YARN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754161#comment-13754161 ] Konstantin Boudnik commented on YARN-696: - bq. For example, you can name you patches YARN-696.1.patch, YARN-696.20130829.patch or with your own ordering method. You don't need to create your own ordering - JIRA does it for. Just keep uploading the patch with the same name and observe the magic ;) Enable multiple states to to be specified in Resource Manager apps REST call Key: YARN-696 URL: https://issues.apache.org/jira/browse/YARN-696 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Assignee: Trevor Lorimer Attachments: YARN-696.diff Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://rm http address:port/ws/v1/cluster/apps). There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7). The proposal is to be able to specify multiple states in a single REST call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out stateless services from stateful services in the RM
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754159#comment-13754159 ] Karthik Kambatla commented on YARN-1098: Currently, there are no Standby services. We might have some in the future depending on how we build HA awareness into the external-facing services. For instance, if we implement a separate HTTPRedirector for redirecting HTTP traffic instead of incorporating in the webapp, HTTPRedirector becomes a Standby service which runs only when the RM is in Standby state. For now, we can start without the Standby-services-CompositeService and add it when we need. For the first cut, we can move everything to ActiveStateServices except the AdminService and RMHAProtocolService. As we implement HA awareness into each of those services, they can be moved to the Always On pool or Standby pool. What do you think? Separate out stateless services from stateful services in the RM Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754167#comment-13754167 ] Jian He commented on YARN-1107: --- bq. I don't want to compare masterKeyId field of the identifier as it will get renewed and updated. Hence added my custom method. The masterKeyId inside the token should not be changed, even though the token is renewed. Only the DelegationKey used to generate the Token will be rolled. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[
https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754168#comment-13754168
]
Hadoop QA commented on YARN-1117:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600671/YARN-1117.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1797//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1797//console
This message is automatically generated.
Improve help message for $ yarn applications and $yarn node
---
Key: YARN-1117
URL: https://issues.apache.org/jira/browse/YARN-1117
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1117.1.patch, YARN-1117.2.patch
There is standardization of help message in YARN-1080. It is nice to have
similar changes for $ yarn appications and yarn node
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[
https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754172#comment-13754172
]
Hadoop QA commented on YARN-981:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600673/YARN-981.4.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1798//console
This message is automatically generated.
YARN/MR2/Job-history /logs link does not have correct content
-
Key: YARN-981
URL: https://issues.apache.org/jira/browse/YARN-981
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Jian He
Fix For: 2.1.1-beta
Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch,
YARN-981.4.patch, YARN-981.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[ https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-981: - Attachment: YARN-981.5.patch Rebased the patch, upload again YARN/MR2/Job-history /logs link does not have correct content - Key: YARN-981 URL: https://issues.apache.org/jira/browse/YARN-981 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch, YARN-981.4.patch, YARN-981.5.patch, YARN-981.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[
https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754189#comment-13754189
]
Hadoop QA commented on YARN-981:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600678/YARN-981.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1799//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1799//console
This message is automatically generated.
YARN/MR2/Job-history /logs link does not have correct content
-
Key: YARN-981
URL: https://issues.apache.org/jira/browse/YARN-981
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Jian He
Fix For: 2.1.1-beta
Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch,
YARN-981.4.patch, YARN-981.5.patch, YARN-981.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1027) Implement RMHAServiceProtocol
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1027: --- Attachment: yarn-1027-in-rm-poc.patch Uploading a proof-of-concept patch that implements HA in RM itself via the RMHAProtocolService. Implement RMHAServiceProtocol - Key: YARN-1027 URL: https://issues.apache.org/jira/browse/YARN-1027 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: yarn-1027-1.patch, yarn-1027-in-rm-poc.patch Implement existing HAServiceProtocol from Hadoop common. This protocol is the single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754209#comment-13754209 ] Xuan Gong commented on YARN-1117: - Return non-zero exit code instead of throwing exceptions. Fix help messages. Improve help message for $ yarn applications and $yarn node --- Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: YARN-1117.1.patch, YARN-1117.2.patch, YARN-1117.3.patch There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[ https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1117: Attachment: YARN-1117.3.patch Improve help message for $ yarn applications and $yarn node --- Key: YARN-1117 URL: https://issues.apache.org/jira/browse/YARN-1117 Project: Hadoop YARN Issue Type: Improvement Components: client Reporter: Tassapol Athiapinya Assignee: Xuan Gong Attachments: YARN-1117.1.patch, YARN-1117.2.patch, YARN-1117.3.patch There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1117) Improve help message for $ yarn applications and $yarn node
[
https://issues.apache.org/jira/browse/YARN-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754238#comment-13754238
]
Hadoop QA commented on YARN-1117:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600689/YARN-1117.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1800//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1800//console
This message is automatically generated.
Improve help message for $ yarn applications and $yarn node
---
Key: YARN-1117
URL: https://issues.apache.org/jira/browse/YARN-1117
Project: Hadoop YARN
Issue Type: Improvement
Components: client
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Attachments: YARN-1117.1.patch, YARN-1117.2.patch, YARN-1117.3.patch
There is standardization of help message in YARN-1080. It is nice to have
similar changes for $ yarn appications and yarn node
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-979) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-979: - Description: ApplicationHistoryProtocol should have the following APIs as well: * getApplicationAttemptReport * getApplicationAttempts * getContainerReport * getContainers The corresponding request and response classes need to be added as well. was: Adding application attempt and container to ApplicationHistoryProtocol Thanks, Mayank Summary: [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol (was: [YARN-321] Adding application attempt and container to ApplicationHistoryProtocol) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol -- Key: YARN-979 URL: https://issues.apache.org/jira/browse/YARN-979 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Zhijie Shen Attachments: YARN-979-1.patch ApplicationHistoryProtocol should have the following APIs as well: * getApplicationAttemptReport * getApplicationAttempts * getContainerReport * getContainers The corresponding request and response classes need to be added as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-979) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-979: - Attachment: YARN-979.2.patch Based on [~mayank_bansal]'s initial patch, I addressed my previous comments and created a new one. The major modifications are: 1. Polishing the javadoc 2. Defining the corresponding protos in application_history_client.proto, yarn_protos.proto and yarn_service_protos.proto 3. Annotate the APIs where it is necessary I've verified that patch can build correctly with branch YARN-321. It is worth mentioning that I've created the empty ApplicationAttemptReport and ContainerReport. They are client oriented. What fields should be put into these two classes will be discussed in separate jiras. In fact, ApplicationAttemptReport has been discussed in YARN-978. Similarly, ContainerReport should be proposed as well because either Container and RMContainer is the good class to be exposed to the client. Will open another jira for it. [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol -- Key: YARN-979 URL: https://issues.apache.org/jira/browse/YARN-979 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Zhijie Shen Attachments: YARN-979-1.patch, YARN-979.2.patch ApplicationHistoryProtocol should have the following APIs as well: * getApplicationAttemptReport * getApplicationAttempts * getContainerReport * getContainers The corresponding request and response classes need to be added as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1123) [YARN-321] Adding ContainerReport and Protobuf implementation
Zhijie Shen created YARN-1123: - Summary: [YARN-321] Adding ContainerReport and Protobuf implementation Key: YARN-1123 URL: https://issues.apache.org/jira/browse/YARN-1123 Project: Hadoop YARN Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Like YARN-978, we need some client-oriented class to expose the container history info. Neither Container nor RMContainer is the right one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-979) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
[
https://issues.apache.org/jira/browse/YARN-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754267#comment-13754267
]
Hadoop QA commented on YARN-979:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600703/YARN-979.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1801//console
This message is automatically generated.
[YARN-321] Add more APIs related to ApplicationAttempt and Container in
ApplicationHistoryProtocol
--
Key: YARN-979
URL: https://issues.apache.org/jira/browse/YARN-979
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Zhijie Shen
Attachments: YARN-979-1.patch, YARN-979.2.patch
ApplicationHistoryProtocol should have the following APIs as well:
* getApplicationAttemptReport
* getApplicationAttempts
* getContainerReport
* getContainers
The corresponding request and response classes need to be added as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754268#comment-13754268 ] Vinod Kumar Vavilapalli commented on YARN-707: -- Looks good to me. I am going to check this in (very nervously).. Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Jason Lowe Priority: Blocker Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-707: - Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) Marking this as an incompatible change. The client-token is for auth between client and AM and clearly MR AM only uses it now. This token is given out for every client separately, so MR 2.1 clients won't have the corresponding information and so will be turned away from 2.1.1 MR AMs. Checked in an incompatible change post 2.1.0 so as to fix MAPREDUCE-5475, a critical issue. Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Jason Lowe Priority: Blocker Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1098) Separate out RM services into Always On and Active
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1098: --- Summary: Separate out RM services into Always On and Active (was: Separate out stateless services from stateful services in the RM) Separate out RM services into Always On and Active -- Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[ https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-981: - Attachment: YARN-981.6.patch Add one more test to test the web service is accessible. YARN/MR2/Job-history /logs link does not have correct content - Key: YARN-981 URL: https://issues.apache.org/jira/browse/YARN-981 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Jian He Fix For: 2.1.1-beta Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch, YARN-981.4.patch, YARN-981.5.patch, YARN-981.6.patch, YARN-981.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1098) Separate out RM services into Always On and Active
[ https://issues.apache.org/jira/browse/YARN-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754291#comment-13754291 ] Karthik Kambatla commented on YARN-1098: Spoke to Bikas and Vinod offline. To begin with, AdminService is the only Always On service. All others go into Active services. As the external services become aware of HA, we move them to the Always On group. Standby group can be created when needed. Separate out RM services into Always On and Active -- Key: YARN-1098 URL: https://issues.apache.org/jira/browse/YARN-1098 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla Assignee: Karthik Kambatla Labels: ha Attachments: yarn-1098-approach.patch, yarn-1098-approach.patch From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby(). The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-707) Add user info in the YARN ClientToken
[ https://issues.apache.org/jira/browse/YARN-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754294#comment-13754294 ] Hudson commented on YARN-707: - SUCCESS: Integrated in Hadoop-trunk-Commit #4351 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4351/]) YARN-707. Added user information also in the YARN ClientToken so that AMs can implement authorization based on incoming users. Contributed by Jason Lowe. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1518868) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenIdentifier.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/ClientToAMTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java Add user info in the YARN ClientToken - Key: YARN-707 URL: https://issues.apache.org/jira/browse/YARN-707 Project: Hadoop YARN Issue Type: Improvement Reporter: Bikas Saha Assignee: Jason Lowe Priority: Blocker Fix For: 3.0.0, 2.1.1-beta Attachments: YARN-707-20130822.txt, YARN-707-20130827.txt, YARN-707-20130828-2.txt, YARN-707-20130828.txt, YARN-707-20130829.txt If user info is present in the client token then it can be used to do limited authz in the AM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED
Omkar Vinit Joshi created YARN-1124: --- Summary: By default yarn application -list should display all the applications in a state other than FINISHED / FAILED Key: YARN-1124 URL: https://issues.apache.org/jira/browse/YARN-1124 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Xuan Gong Priority: Blocker Fix For: 2.1.1-beta Today we are just listing application in RUNNING state by default for yarn application -list. Instead we should show all the applications which are either submitted/accepted/running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-981) YARN/MR2/Job-history /logs link does not have correct content
[
https://issues.apache.org/jira/browse/YARN-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754304#comment-13754304
]
Hadoop QA commented on YARN-981:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600708/YARN-981.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1802//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1802//console
This message is automatically generated.
YARN/MR2/Job-history /logs link does not have correct content
-
Key: YARN-981
URL: https://issues.apache.org/jira/browse/YARN-981
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Jian He
Fix For: 2.1.1-beta
Attachments: YARN-981.1.patch, YARN-981.2.patch, YARN-981.3.patch,
YARN-981.4.patch, YARN-981.5.patch, YARN-981.6.patch, YARN-981.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAServiceProtocol
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754306#comment-13754306 ] Karthik Kambatla commented on YARN-1027: Discussion with Bikas and Vinod offline: The HA-in-RM approach doesn't seem to be much more disruptive than the wrapper/extension approaches, particularly given the changes in YARN-1098. The implementation can be along the lines of the proof-of-concept patch. Implement RMHAServiceProtocol - Key: YARN-1027 URL: https://issues.apache.org/jira/browse/YARN-1027 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Karthik Kambatla Attachments: yarn-1027-1.patch, yarn-1027-in-rm-poc.patch Implement existing HAServiceProtocol from Hadoop common. This protocol is the single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1125) Add shutdown support to non-service RM components
Karthik Kambatla created YARN-1125: -- Summary: Add shutdown support to non-service RM components Key: YARN-1125 URL: https://issues.apache.org/jira/browse/YARN-1125 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Karthik Kambatla The ResourceManager has certain non-service components like the Scheduler. While transitioning to standby, these components should be completely turned off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754319#comment-13754319 ] Omkar Vinit Joshi commented on YARN-1107: - updating the patch. Reordering the services to make sure that RMDelegationTokenRenewer does not start before ClientRMService. Also all the tokens added for renewal will be enqueued until the service moves into STARTED state. As a part of service start it will renew all the tokens. Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[ https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-1107: Attachment: YARN-1107.20130829.1.patch Job submitted with Delegation token in secured environment causes RM to fail during RM restart -- Key: YARN-1107 URL: https://issues.apache.org/jira/browse/YARN-1107 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta Assignee: Omkar Vinit Joshi Priority: Blocker Attachments: rm.log, YARN-1107.20130828.1.patch, YARN-1107.20130829.1.patch If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1107) Job submitted with Delegation token in secured environment causes RM to fail during RM restart
[
https://issues.apache.org/jira/browse/YARN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754331#comment-13754331
]
Hadoop QA commented on YARN-1107:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600718/YARN-1107.20130829.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1803//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1803//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1803//console
This message is automatically generated.
Job submitted with Delegation token in secured environment causes RM to fail
during RM restart
--
Key: YARN-1107
URL: https://issues.apache.org/jira/browse/YARN-1107
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Omkar Vinit Joshi
Priority: Blocker
Attachments: rm.log, YARN-1107.20130828.1.patch,
YARN-1107.20130829.1.patch
If secure RM with recovery enabled is restarted while oozie jobs are running
rm fails to come up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[
https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754336#comment-13754336
]
Junping Du commented on YARN-771:
-
Hi [~bikassaha], Thanks for good comments above.
I have different ideas here that I thought every results are determinate so far.
case 1: app add node-a to addition list and then add it to removal list (tends
to remove it).
{code}
client.updateBlacklist(a, null);
client.updateBlacklist(null, a);
client.schedule();
//{a,a} will be sent, and a will not in blacklist
{code}
case 2: app add node-a to removal list and then add node-a to addition list
(tends to add node-a back to blacklist)
{code}
client.updateBlacklist(null, a);
client.updateBlacklist(a, null);
client.schedule();
//(a, null) will be sent, so a will be in blacklist
{code}
case 3: app add node-a to removal list and addition list at the same call.
{code}
client.updateBlacklist(a, a);
client.schedule();
//{a, a} will be sent and a will not in blacklist
{code}
I understand there are different opinions around case 3 that it leads to user
confused to mean I want node-a in blacklist or I want node-a not in
blacklist. My question is: can this confusion be resolved through proper
documentation?
Map to real cases, the previous one may come from case which seems rare to me -
app said I want to remove nodes from blacklist in set R except it is in set A
(assume node-a in both set A and set R). But the latter one may come from more
often case - app said I want nodes from set A go to blacklist except it is in
set R (node-a in both sets). We can better support latter case here rather
than updateBlacklist twice. Thoughts?
AMRMClient support for resource blacklisting
-
Key: YARN-771
URL: https://issues.apache.org/jira/browse/YARN-771
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Junping Du
Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, YARN-771-v3.patch
After YARN-750 AMRMClient should support blacklisting via the new YARN API's
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED
[ https://issues.apache.org/jira/browse/YARN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-1124: Attachment: YARN-1124.1.patch By default yarn application -list should display all the applications in a state other than FINISHED / FAILED - Key: YARN-1124 URL: https://issues.apache.org/jira/browse/YARN-1124 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Xuan Gong Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1124.1.patch Today we are just listing application in RUNNING state by default for yarn application -list. Instead we should show all the applications which are either submitted/accepted/running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED
[ https://issues.apache.org/jira/browse/YARN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754356#comment-13754356 ] Xuan Gong commented on YARN-1124: - listing submitted/accepted as well as RUNNING state by default for yarn application -list modify the test cases to test it By default yarn application -list should display all the applications in a state other than FINISHED / FAILED - Key: YARN-1124 URL: https://issues.apache.org/jira/browse/YARN-1124 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Xuan Gong Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1124.1.patch Today we are just listing application in RUNNING state by default for yarn application -list. Instead we should show all the applications which are either submitted/accepted/running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1126) Add validation of users input nodes-states options to nodes CLI
Wei Yan created YARN-1126: - Summary: Add validation of users input nodes-states options to nodes CLI Key: YARN-1126 URL: https://issues.apache.org/jira/browse/YARN-1126 Project: Hadoop YARN Issue Type: Improvement Reporter: Wei Yan Assignee: Wei Yan Follow the discussion in YARN-905. (1) case-insensitive checks for all. (2) validation of users input, exit with non-zero code and print all valid states when user gives an invalid state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-905) Add state filters to nodes CLI
[ https://issues.apache.org/jira/browse/YARN-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754362#comment-13754362 ] Wei Yan commented on YARN-905: -- [~vinodkv][~sandyr] YARN-1126 is created. Add state filters to nodes CLI -- Key: YARN-905 URL: https://issues.apache.org/jira/browse/YARN-905 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Wei Yan Attachments: Yarn-905.patch, YARN-905.patch, YARN-905.patch It would be helpful for the nodes CLI to have a node-states option that allows it to return nodes that are not just in the RUNNING state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED
[
https://issues.apache.org/jira/browse/YARN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754370#comment-13754370
]
Hadoop QA commented on YARN-1124:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600724/YARN-1124.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1804//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1804//console
This message is automatically generated.
By default yarn application -list should display all the applications in a
state other than FINISHED / FAILED
-
Key: YARN-1124
URL: https://issues.apache.org/jira/browse/YARN-1124
Project: Hadoop YARN
Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Xuan Gong
Priority: Blocker
Fix For: 2.1.1-beta
Attachments: YARN-1124.1.patch
Today we are just listing application in RUNNING state by default for yarn
application -list. Instead we should show all the applications which are
either submitted/accepted/running.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-771) AMRMClient support for resource blacklisting
[ https://issues.apache.org/jira/browse/YARN-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754397#comment-13754397 ] Bikas Saha commented on YARN-771: - The cases listed above are possible. However, IMO, an API should be clear by itself. It should be clear in a programmatic way. Secondly, the AMRMClient should make sure its doing the correct thing on its own instead of sending something inconsistent/incorrect to the RM and depending on RM to fix things. We are fine with the current RM logic but RM internal logic may change on its own cycle and break the assumptions made here. We should always send the last decision of the client to the RM. Consider the following case. Say, RM has already been notified about A in blacklist. Now if user adds A to blacklist again then removes A from blacklist (case 1 above) then users final decision is to remove A. If we resolve this locally and clear A from both lists locally then the users last decision will not be sent to the RM. Hence RM will continue to have A in its blacklist. Hence, we should always make sure that the users last decision is honored and sent to the RM. The issue arises when user adds and removes the same node in the same invocation. We can either choose to 1) throw an IllegalArgumentException or 2) log a warning message and send the removal to the RM. Sending removal is safe since if the node was already blacklisted on the RM then it should be removed. If was not already blacklisted on the RM then removing it should be a no-op. AMRMClient support for resource blacklisting - Key: YARN-771 URL: https://issues.apache.org/jira/browse/YARN-771 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Junping Du Attachments: YARN-771-v1.0.patch, YARN-771-v2.patch, YARN-771-v3.patch After YARN-750 AMRMClient should support blacklisting via the new YARN API's -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1124) By default yarn application -list should display all the applications in a state other than FINISHED / FAILED
[ https://issues.apache.org/jira/browse/YARN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13754398#comment-13754398 ] Abhishek Kapoor commented on YARN-1124: --- Pardon my ignorance. I don't see why we have to list submitted/accepted/running application. Isn't that a job of job-History server ? By default yarn application -list should display all the applications in a state other than FINISHED / FAILED - Key: YARN-1124 URL: https://issues.apache.org/jira/browse/YARN-1124 Project: Hadoop YARN Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Xuan Gong Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1124.1.patch Today we are just listing application in RUNNING state by default for yarn application -list. Instead we should show all the applications which are either submitted/accepted/running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
