date:20140226


[ 
https://issues.apache.org/jira/browse/YARN-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912763#comment-13912763
 ] 

Hadoop QA commented on YARN-1752:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631191/YARN-1752.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3189//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3189//console

This message is automatically generated.

 Unexpected Unregistered event at Attempt Launched state
 ---

 Key: YARN-1752
 URL: https://issues.apache.org/jira/browse/YARN-1752
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
 Attachments: YARN-1752.1.patch


 {code}
 2014-02-21 14:56:03,453 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
 Can't handle this event at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 UNREGISTERED at LAUNCHED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
   at java.lang.Thread.run(Thread.java:695)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1561) Fix a generic type warning in FairScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912769#comment-13912769
 ] 

Hudson commented on YARN-1561:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #493 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/493/])
YARN-1561. Fix a generic type warning in FairScheduler. (Chen He via 
junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571924)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java


 Fix a generic type warning in FairScheduler
 ---

 Key: YARN-1561
 URL: https://issues.apache.org/jira/browse/YARN-1561
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Reporter: Junping Du
Assignee: Chen He
Priority: Minor
  Labels: newbie
 Fix For: 2.5.0

 Attachments: yarn-1561.patch


 The Comparator below should be specified with type:
 private Comparator nodeAvailableResourceComparator =
   new NodeAvailableResourceComparator(); 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1760) TestRMAdminService assumes CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912765#comment-13912765
 ] 

Hudson commented on YARN-1760:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #493 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/493/])
YARN-1760. TestRMAdminService assumes CapacityScheduler. (kasha) (kasha: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571777)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java


 TestRMAdminService assumes CapacityScheduler
 

 Key: YARN-1760
 URL: https://issues.apache.org/jira/browse/YARN-1760
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: test
 Fix For: 2.4.0

 Attachments: yarn-1760-1.patch, yarn-1760-2.patch, yarn-1760-3.patch


 YARN-1611 adds TestRMAdminService which assumes the use of CapacityScheduler. 
 {noformat}
 java.lang.ClassCastException: 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 
 cannot be cast to 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testAdminRefreshQueuesWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:115)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1760) TestRMAdminService assumes CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912924#comment-13912924
 ] 

Hudson commented on YARN-1760:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1685 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1685/])
YARN-1760. TestRMAdminService assumes CapacityScheduler. (kasha) (kasha: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571777)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java


 TestRMAdminService assumes CapacityScheduler
 

 Key: YARN-1760
 URL: https://issues.apache.org/jira/browse/YARN-1760
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: test
 Fix For: 2.4.0

 Attachments: yarn-1760-1.patch, yarn-1760-2.patch, yarn-1760-3.patch


 YARN-1611 adds TestRMAdminService which assumes the use of CapacityScheduler. 
 {noformat}
 java.lang.ClassCastException: 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 
 cannot be cast to 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testAdminRefreshQueuesWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:115)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1561) Fix a generic type warning in FairScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912928#comment-13912928
 ] 

Hudson commented on YARN-1561:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1685 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1685/])
YARN-1561. Fix a generic type warning in FairScheduler. (Chen He via 
junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571924)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java


 Fix a generic type warning in FairScheduler
 ---

 Key: YARN-1561
 URL: https://issues.apache.org/jira/browse/YARN-1561
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Reporter: Junping Du
Assignee: Chen He
Priority: Minor
  Labels: newbie
 Fix For: 2.5.0

 Attachments: yarn-1561.patch


 The Comparator below should be specified with type:
 private Comparator nodeAvailableResourceComparator =
   new NodeAvailableResourceComparator(); 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1561) Fix a generic type warning in FairScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912981#comment-13912981
 ] 

Hudson commented on YARN-1561:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1710 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1710/])
YARN-1561. Fix a generic type warning in FairScheduler. (Chen He via 
junping_du) (junping_du: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571924)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java


 Fix a generic type warning in FairScheduler
 ---

 Key: YARN-1561
 URL: https://issues.apache.org/jira/browse/YARN-1561
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Reporter: Junping Du
Assignee: Chen He
Priority: Minor
  Labels: newbie
 Fix For: 2.5.0

 Attachments: yarn-1561.patch


 The Comparator below should be specified with type:
 private Comparator nodeAvailableResourceComparator =
   new NodeAvailableResourceComparator(); 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1760) TestRMAdminService assumes CapacityScheduler


[ 
https://issues.apache.org/jira/browse/YARN-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13912977#comment-13912977
 ] 

Hudson commented on YARN-1760:
--

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1710 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1710/])
YARN-1760. TestRMAdminService assumes CapacityScheduler. (kasha) (kasha: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1571777)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAdminService.java


 TestRMAdminService assumes CapacityScheduler
 

 Key: YARN-1760
 URL: https://issues.apache.org/jira/browse/YARN-1760
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Trivial
  Labels: test
 Fix For: 2.4.0

 Attachments: yarn-1760-1.patch, yarn-1760-2.patch, yarn-1760-3.patch


 YARN-1611 adds TestRMAdminService which assumes the use of CapacityScheduler. 
 {noformat}
 java.lang.ClassCastException: 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler 
 cannot be cast to 
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
   at 
 org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testAdminRefreshQueuesWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:115)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1752) Unexpected Unregistered event at Attempt Launched state

2014-02-26 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913088#comment-13913088
 ] 

Hitesh Shah commented on YARN-1752:
---

[~rohithsharma] The fix should be in YARN - by fixing MR, you are just hiding 
the problem and not fixing it.

As for the MR AM behaving correctly, you can create a separate jira under 
MapReduce and provided the current patch there.

 Unexpected Unregistered event at Attempt Launched state
 ---

 Key: YARN-1752
 URL: https://issues.apache.org/jira/browse/YARN-1752
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
 Attachments: YARN-1752.1.patch


 {code}
 2014-02-21 14:56:03,453 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
 Can't handle this event at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 UNREGISTERED at LAUNCHED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
   at java.lang.Thread.run(Thread.java:695)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


 [ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1713:


Attachment: apache-yarn-1713.cumulative.2.patch

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.patch, 
 apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


 [ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1713:


Attachment: apache-yarn-1713.3.patch

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.patch, 
 apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1702) Expose kill app functionality as part of RM web services


 [ 
https://issues.apache.org/jira/browse/YARN-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1702:


Attachment: apache-yarn-1702.7.patch

 Expose kill app functionality as part of RM web services
 

 Key: YARN-1702
 URL: https://issues.apache.org/jira/browse/YARN-1702
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1702.2.patch, apache-yarn-1702.3.patch, 
 apache-yarn-1702.4.patch, apache-yarn-1702.5.patch, apache-yarn-1702.7.patch


 Expose functionality to kill an app via the ResourceManager web services API.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


[ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913187#comment-13913187
 ] 

Varun Vasudev commented on YARN-1713:
-

 styling issue, please follow the convention of 80 column limit
Fixed the styling issue

 appIdToRMApp is not actually adding Id to RMApp, more likely 
 getRMAppFromRMContext() ?
Yes. Renamed to getRMAppForAppId

 we have created factory method in each user-facing record for instantiating 
 the record, e.g.: ApplicationSubmissionContext.newInstance, you can use that.
Fixed

 createNewApplication should be a get request as it only returns the 
 applicationId etc. just like the one in ClientRMService.getNewApplication
I'm not sure about this. According to REST principles, GET is supposed to 
idempotent. Is getNewApplication idempotent?

 you can attach a name for the XmlRootElement like @XmlRootElement(name = 
 appAttempt) to specify the element name, so we can do @XmlRootElement(name 
 = “newApplication”)
Fixed

 Note that RMWebServices.hasAcess() is checked against VIEW_APP permission, in 
 the case of submit/kill, we should check against MODIFY_APP
Fixed


 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.patch, 
 apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission


[ 
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913212#comment-13913212
 ] 

Xuan Gong commented on YARN-1410:
-

This testcase failure is un-related

 Handle client failover during 2 step client API's like app submission
 -

 Key: YARN-1410
 URL: https://issues.apache.org/jira/browse/YARN-1410
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, 
 YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, 
 YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 App submission involves
 1) creating appId
 2) using that appId to submit an ApplicationSubmissionContext to the user.
 The client may have obtained an appId from an RM, the RM may have failed 
 over, and the client may submit the app to the new RM.
 Since the new RM has a different notion of cluster timestamp (used to create 
 app id) the new RM may reject the app submission resulting in unexpected 
 failure on the client side.
 The same may happen for other 2 step client API operations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


[ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913241#comment-13913241
 ] 

Hadoop QA commented on YARN-1713:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12631283/apache-yarn-1713.cumulative.2.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 2 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3190//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3190//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3190//console

This message is automatically generated.

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.patch, 
 apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


 [ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1713:


Attachment: apache-yarn-1713.4.patch

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.patch, 
 apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


 [ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-1713:


Attachment: apache-yarn-1713.cumulative.3.patch

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.3.patch, 
 apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking


 [ 
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-1363:
--

Attachment: YARN-1363.8.patch

Create a new patch, which changes the solution. Instead of making the 
RMStateStore operate token asynchronously, I created a 
RMDelegationTokenSecretManagerAsync which receive the operation events from 
ClientRMService, handle them one by one on a separate thread. It can the RPC 
call unblocked as well as keeping the change modular, though the size of patch 
is not reduced. Another important change in this patch is that instead of 
touching the hadoop common code base, I created separate get/cancel/renew 
response for yarn project only.

 Get / Cancel / Renew delegation token api should be non blocking
 

 Key: YARN-1363
 URL: https://issues.apache.org/jira/browse/YARN-1363
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
 Attachments: YARN-1363.1.patch, YARN-1363.2.patch, YARN-1363.3.patch, 
 YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, 
 YARN-1363.8.patch


 Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are 
 all blocking apis.
 * As a part of these calls we try to update RMStateStore and that may slow it 
 down.
 * Now as we have limited number of client request handlers we may fill up 
 client handlers quickly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


[ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913326#comment-13913326
 ] 

Hadoop QA commented on YARN-1713:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12631301/apache-yarn-1713.cumulative.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3191//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3191//console

This message is automatically generated.

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.3.patch, 
 apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking


 [ 
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-1363:
--

Attachment: YARN-1363.9.patch

Add one more test case to verify RMDelegationTokenSecretManagerAsync.

 Get / Cancel / Renew delegation token api should be non blocking
 

 Key: YARN-1363
 URL: https://issues.apache.org/jira/browse/YARN-1363
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
 Attachments: YARN-1363.1.patch, YARN-1363.2.patch, YARN-1363.3.patch, 
 YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, 
 YARN-1363.8.patch, YARN-1363.9.patch


 Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are 
 all blocking apis.
 * As a part of these calls we try to update RMStateStore and that may slow it 
 down.
 * Now as we have limited number of client request handlers we may fill up 
 client handlers quickly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1730) Leveldb timeline store needs simple write locking


[ 
https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913377#comment-13913377
 ] 

Zhijie Shen commented on YARN-1730:
---

The latest patch looks good to me overall. Just one more suggestion. Is it good 
to make the cache size configurable?

 Leveldb timeline store needs simple write locking
 -

 Key: YARN-1730
 URL: https://issues.apache.org/jira/browse/YARN-1730
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, 
 YARN-1730.4.patch


 The actual data writes are performed atomically in a batch, but a lock should 
 be held while identifying a start time for the entity, which precedes every 
 write.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


[ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913438#comment-13913438
 ] 

Vinod Kumar Vavilapalli commented on YARN-1588:
---

Looks good, +1. Checking this in.

 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch, 
 YARN-1588.3.patch, YARN-1588.4.patch, YARN-1588.5.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking


[ 
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913447#comment-13913447
 ] 

Hadoop QA commented on YARN-1363:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631303/YARN-1363.8.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
  
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3192//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3192//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-YARN-Build/3192//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3192//console

This message is automatically generated.

 Get / Cancel / Renew delegation token api should be non blocking
 

 Key: YARN-1363
 URL: https://issues.apache.org/jira/browse/YARN-1363
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
 Attachments: YARN-1363.1.patch, YARN-1363.2.patch, YARN-1363.3.patch, 
 YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch, 
 YARN-1363.8.patch, YARN-1363.9.patch


 Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are 
 all blocking apis.
 * As a part of these calls we try to update RMStateStore and that may slow it 
 down.
 * Now as we have limited number of client request handlers we may fill up 
 client handlers quickly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1588) Rebind NM tokens for previous attempt's running containers to the new attempt


[ 
https://issues.apache.org/jira/browse/YARN-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913474#comment-13913474
 ] 

Hudson commented on YARN-1588:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5233 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5233/])
YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of 
transferred containers from previous app-attempts to new AMs after YARN-1490. 
Contributed by Jian He. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1572230)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java


 Rebind NM tokens for previous attempt's running containers to the new attempt
 -

 Key: YARN-1588
 URL: https://issues.apache.org/jira/browse/YARN-1588
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
 Fix For: 2.4.0

 Attachments: YARN-1588.1.patch, YARN-1588.1.patch, YARN-1588.2.patch, 
 YARN-1588.3.patch, YARN-1588.4.patch, YARN-1588.5.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1490) RM should optionally not kill all containers when an ApplicationMaster exits


[ 
https://issues.apache.org/jira/browse/YARN-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913473#comment-13913473
 ] 

Hudson commented on YARN-1490:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5233 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5233/])
YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of 
transferred containers from previous app-attempts to new AMs after YARN-1490. 
Contributed by Jian He. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1572230)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterResponse.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NMToken.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/RegisterApplicationMasterResponsePBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/NMTokenPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java


 RM should optionally not kill all containers when an ApplicationMaster exits
 

 Key: YARN-1490
 URL: https://issues.apache.org/jira/browse/YARN-1490
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Jian He
 Fix For: 2.4.0

 Attachments: YARN-1490.1.patch, YARN-1490.10.patch, 
 YARN-1490.11.patch, YARN-1490.11.patch, YARN-1490.12.patch, 
 YARN-1490.2.patch, YARN-1490.3.patch, YARN-1490.4.patch, YARN-1490.5.patch, 
 YARN-1490.6.patch, YARN-1490.7.patch, YARN-1490.8.patch, YARN-1490.9.patch, 
 org.apache.oozie.service.TestRecoveryService_thread-dump.txt


 This is needed to enable work-preserving AM restart. Some apps can chose to 
 reconnect with old running containers, some may not want to. This should be 
 an option.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1685) Log URL should be different when the container is running and finished, and null case needs to be handled


[ 
https://issues.apache.org/jira/browse/YARN-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913478#comment-13913478
 ] 

Zhijie Shen commented on YARN-1685:
---

Link the comment in YARN-1700: 
https://issues.apache.org/jira/browse/YARN-1700?focusedCommentId=13912551page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13912551

Let's make sure the log url is pointing to NM web page when the container is 
running, and to AHS when it is finished

 Log URL should be different when the container is running and finished, and 
 null case needs to be handled
 -

 Key: YARN-1685
 URL: https://issues.apache.org/jira/browse/YARN-1685
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Mayank Bansal
 Fix For: YARN-321

 Attachments: YARN-1685-1.patch


 https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866416page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866416
 https://issues.apache.org/jira/browse/YARN-1413?focusedCommentId=13866844page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13866844



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (YARN-1766) When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.

Xuan Gong created YARN-1766:
---

 Summary: When RM does the initiation, it should use loaded 
Configuration instead of bootstrap configuration.
 Key: YARN-1766
 URL: https://issues.apache.org/jira/browse/YARN-1766
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong


Right now, we have FileSystemBasedConfigurationProvider to let Users upload the 
configurations into remote File System, and let different RMs share the same 
configurations.  During the initiation, RM will load the configurations from 
Remote File System. So when RM initiates the services, it should use the loaded 
Configurations instead of using the bootstrap configurations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking

[
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913528#comment-13913528
]

Hadoop QA commented on YARN-1363:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631320/YARN-1363.9.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.

{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.

{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.mapreduce.v2.TestUberAM

org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService

{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3193//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3193//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/3193//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3193//console

This message is automatically generated.

Get / Cancel / Renew delegation token api should be non blocking

Key: YARN-1363
URL: https://issues.apache.org/jira/browse/YARN-1363
Project: Hadoop YARN
Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
Attachments: YARN-1363.1.patch, YARN-1363.2.patch, YARN-1363.3.patch,
YARN-1363.4.patch, YARN-1363.5.patch, YARN-1363.6.patch, YARN-1363.7.patch,
YARN-1363.8.patch, YARN-1363.9.patch

Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are
all blocking apis.
* As a part of these calls we try to update RMStateStore and that may slow it
down.
* Now as we have limited number of client request handlers we may fill up
client handlers quickly.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1542) Add unit test for public resource on viewfs

2014-02-26 Thread Gera Shegalov (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913542#comment-13913542
 ] 

Gera Shegalov commented on YARN-1542:
-

[~vinodkv], please take a look at my original 
[comment|https://issues.apache.org/jira/browse/YARN-1542?focusedCommentId=13857255page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13857255]
 for the root cause of the failure and why I am adding this test.

 Add unit test for public resource on viewfs
 ---

 Key: YARN-1542
 URL: https://issues.apache.org/jira/browse/YARN-1542
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Reporter: Gera Shegalov
Assignee: Gera Shegalov
 Attachments: YARN-1542.v01.patch, YARN-1542.v02.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1730) Leveldb timeline store needs simple write locking

2014-02-26 Thread Billie Rinaldi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-1730:
-

Attachment: YARN-1730.5.patch

Added configurable cache sizes.  Let me know how it looks.

 Leveldb timeline store needs simple write locking
 -

 Key: YARN-1730
 URL: https://issues.apache.org/jira/browse/YARN-1730
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, 
 YARN-1730.4.patch, YARN-1730.5.patch


 The actual data writes are performed atomically in a batch, but a lock should 
 be held while identifying a start time for the entity, which precedes every 
 write.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1410) Handle client failover during 2 step client API's like app submission

2014-02-26 Thread Bikas Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913612#comment-13913612
 ] 

Bikas Saha commented on YARN-1410:
--

Can we do this in the RM? Then if we are going to retry the submitApplication 
RPC we might be better off. The response of a successful submitApplication() 
gives us the applicationId to be used to do the step 2 getApplicationReport. 
This also saves 1 RPC hop.
{code}
 ApplicationId applicationId = appContext.getApplicationId();
-appContext.setApplicationId(applicationId);
+if (applicationId == null) {
+  applicationId = getNewApplication().getApplicationId();
+  appContext.setApplicationId(applicationId);
+}
{code}

Please put some comments in the test to help understand what is being tested. 
e.g. testing that failed over RM accepts the appId in submitContext even though 
it does not exist internally.

If the test is doing failover in the test method then why is 
submitApplication(ApplicationId oldAppId) also causing failover?

Looks like the RM already blindly accepts the appId present in the 
submitContext.

Please change the title of the jira and link it to the other 2 jiras.


 Handle client failover during 2 step client API's like app submission
 -

 Key: YARN-1410
 URL: https://issues.apache.org/jira/browse/YARN-1410
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, 
 YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, 
 YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 App submission involves
 1) creating appId
 2) using that appId to submit an ApplicationSubmissionContext to the user.
 The client may have obtained an appId from an RM, the RM may have failed 
 over, and the client may submit the app to the new RM.
 Since the new RM has a different notion of cluster timestamp (used to create 
 app id) the new RM may reject the app submission resulting in unexpected 
 failure on the client side.
 The same may happen for other 2 step client API operations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1766) When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.


[ 
https://issues.apache.org/jira/browse/YARN-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913630#comment-13913630
 ] 

Xuan Gong commented on YARN-1766:
-

* Upload a patch to fix this issue.
* Also change the order of loading configurations to load core-site.xml first, 
then yarn-site.xml

 When RM does the initiation, it should use loaded Configuration instead of 
 bootstrap configuration.
 ---

 Key: YARN-1766
 URL: https://issues.apache.org/jira/browse/YARN-1766
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong

 Right now, we have FileSystemBasedConfigurationProvider to let Users upload 
 the configurations into remote File System, and let different RMs share the 
 same configurations.  During the initiation, RM will load the configurations 
 from Remote File System. So when RM initiates the services, it should use the 
 loaded Configurations instead of using the bootstrap configurations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493

2014-02-26 Thread Naren Koneru (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913633#comment-13913633
]

Naren Koneru commented on YARN-1577:

After digging through the details, here's the summary as I understand (sorry
for the repetition if any).

- Today, the unmanaged client (llama) is sending a request to launch the AM,
then waiting for the App state to be ACCEPTED and then its registering the AM
using AMRMClientAsync.registerApplicationMaster.
- This register call expects the AM RM token to be set, which is part of the
application report. The Client gets this token by calling
ApplicationClientProtocol.getApplicationReport after the APP is accepted.

With the change in YARN-1493, this is broken since the AppAttempt is launched
after the application is accepted and hence the token is not set. So the client
can run into race conditions depending on when its getting the application
report. The temporary hack we made in the client is to retry for a fixed number
of times.

One way to solve this could be:
- Change the ApplicationReport (returned by
ApplicationClientProtocol.getApplicationReport) to add an attempt state, so the
client can rely on the Attempt state to be launched before proceeding with the
UAM registration.

- However, this would not be backwards compatible since it involves changes to
the unmanaged clients. Since I do not see any documentation for the unmanaged
clients, is this acceptable?

Is this proposal ok?. If not, any other suggestions?. If this proposal is ok,
then I can submit a patch. Pls comment.

Unmanaged AM is broken because of YARN-1493
---

Key: YARN-1577
URL: https://issues.apache.org/jira/browse/YARN-1577
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Jian He
Assignee: Naren Koneru
Priority: Blocker

Today unmanaged AM client is waiting for app state to be Accepted to launch
the AM. This is broken since we changed in YARN-1493 to start the attempt
after the application is Accepted. We may need to introduce an attempt state
report that client can rely on to query the attempt state and choose to
launch the unmanaged AM.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1766) When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.


 [ 
https://issues.apache.org/jira/browse/YARN-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-1766:


Attachment: YARN-1766.1.patch

 When RM does the initiation, it should use loaded Configuration instead of 
 bootstrap configuration.
 ---

 Key: YARN-1766
 URL: https://issues.apache.org/jira/browse/YARN-1766
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
 Attachments: YARN-1766.1.patch


 Right now, we have FileSystemBasedConfigurationProvider to let Users upload 
 the configurations into remote File System, and let different RMs share the 
 same configurations.  During the initiation, RM will load the configurations 
 from Remote File System. So when RM initiates the services, it should use the 
 loaded Configurations instead of using the bootstrap configurations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1730) Leveldb timeline store needs simple write locking


[ 
https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913636#comment-13913636
 ] 

Hadoop QA commented on YARN-1730:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631363/YARN-1730.5.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3194//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3194//console

This message is automatically generated.

 Leveldb timeline store needs simple write locking
 -

 Key: YARN-1730
 URL: https://issues.apache.org/jira/browse/YARN-1730
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, 
 YARN-1730.4.patch, YARN-1730.5.patch


 The actual data writes are performed atomically in a batch, but a lock should 
 be held while identifying a start time for the entity, which precedes every 
 write.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().


 [ 
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-1410:


Summary: Handle RM fails over after getApplicationID() and before 
submitApplication().  (was: Handle client failover during 2 step client API's 
like app submission)

 Handle RM fails over after getApplicationID() and before submitApplication().
 -

 Key: YARN-1410
 URL: https://issues.apache.org/jira/browse/YARN-1410
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, 
 YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, 
 YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 App submission involves
 1) creating appId
 2) using that appId to submit an ApplicationSubmissionContext to the user.
 The client may have obtained an appId from an RM, the RM may have failed 
 over, and the client may submit the app to the new RM.
 Since the new RM has a different notion of cluster timestamp (used to create 
 app id) the new RM may reject the app submission resulting in unexpected 
 failure on the client side.
 The same may happen for other 2 step client API operations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy


 [ 
https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-1740:
--

Attachment: YARN-1740.3.patch

Added one more test to verify the redirected URL scheme is consistently with 
the provided yarn http configuration

 Redirection from AM-URL is broken with HTTPS_ONLY policy
 

 Key: YARN-1740
 URL: https://issues.apache.org/jira/browse/YARN-1740
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Yesha Vora
Assignee: Jian He
 Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch


 Steps to reproduce:
 1) Run a sleep job
 2) Run: yarn application -list command to find AM URL.
 root@host1:~# yarn application -list
 Total number of applications (application-types: [] and states: SUBMITTED, 
 ACCEPTED, RUNNING):1
 Application-Id Application-Name Application-Type User Queue State Final-State 
 Progress Tracking-URL
 application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING 
 UNDEFINED 5% http://host1:40653
 3) Try to access http://host1:40653/ws/v1/mapreduce/info; url.
 This URL redirects to 
 http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
 Here, Http protocol is used with HTTPS port for RM.
 The expected Url is 
 https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1766) When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.


[ 
https://issues.apache.org/jira/browse/YARN-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913726#comment-13913726
 ] 

Hadoop QA commented on YARN-1766:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631373/YARN-1766.1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3195//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3195//console

This message is automatically generated.

 When RM does the initiation, it should use loaded Configuration instead of 
 bootstrap configuration.
 ---

 Key: YARN-1766
 URL: https://issues.apache.org/jira/browse/YARN-1766
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
 Attachments: YARN-1766.1.patch


 Right now, we have FileSystemBasedConfigurationProvider to let Users upload 
 the configurations into remote File System, and let different RMs share the 
 same configurations.  During the initiation, RM will load the configurations 
 from Remote File System. So when RM initiates the services, it should use the 
 loaded Configurations instead of using the bootstrap configurations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate

2014-02-26 Thread Tsuyoshi OZAWA (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated YARN-1301:
-

Attachment: YARN-1301.6.patch

Rebased a patch on trunk. [~zjshen], ListString#toString() shows the contents 
of list like [host1, host2, host3]. I think it's no problem. What do you think?

 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.4.0

 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate


 [ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhijie Shen updated YARN-1301:
--

Target Version/s: 2.4.0
   Fix Version/s: (was: 2.4.0)

 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking


[ 
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913729#comment-13913729
 ] 

Zhijie Shen commented on YARN-1363:
---

TestUberAM failure is not related, and happens on trunk as well. Filed a 
separate ticket to trace it: MAPREDUCE-5768

 Get / Cancel / Renew delegation token api should be non blocking
 

 Key: YARN-1363
 URL: https://issues.apache.org/jira/browse/YARN-1363
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
 Attachments: YARN-1363.1.patch, YARN-1363.10.patch, 
 YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, YARN-1363.5.patch, 
 YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, YARN-1363.9.patch


 Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are 
 all blocking apis.
 * As a part of these calls we try to update RMStateStore and that may slow it 
 down.
 * Now as we have limited number of client request handlers we may fill up 
 client handlers quickly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy


[ 
https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913737#comment-13913737
 ] 

Hadoop QA commented on YARN-1740:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631378/YARN-1740.3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3196//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3196//console

This message is automatically generated.

 Redirection from AM-URL is broken with HTTPS_ONLY policy
 

 Key: YARN-1740
 URL: https://issues.apache.org/jira/browse/YARN-1740
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Yesha Vora
Assignee: Jian He
 Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch


 Steps to reproduce:
 1) Run a sleep job
 2) Run: yarn application -list command to find AM URL.
 root@host1:~# yarn application -list
 Total number of applications (application-types: [] and states: SUBMITTED, 
 ACCEPTED, RUNNING):1
 Application-Id Application-Name Application-Type User Queue State Final-State 
 Progress Tracking-URL
 application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING 
 UNDEFINED 5% http://host1:40653
 3) Try to access http://host1:40653/ws/v1/mapreduce/info; url.
 This URL redirects to 
 http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
 Here, Http protocol is used with HTTPS port for RM.
 The expected Url is 
 https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate


[ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913741#comment-13913741
 ] 

Zhijie Shen commented on YARN-1301:
---

LGTM, will commit it after jenkins +1

 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1730) Leveldb timeline store needs simple write locking


[ 
https://issues.apache.org/jira/browse/YARN-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913739#comment-13913739
 ] 

Vinod Kumar Vavilapalli commented on YARN-1730:
---

More comments
 - Similar to above renames, rename {{getRead|WriteCacheSize()}} APIs to 
include the key-word start-time?
 - Create a prefix constant for *.leveldb-timeline-store in YarnConfiguration.
 - Why do we need separate start-time caches for read and write calls?
 - Shouldn't we do the write-locking at the {{put(..)}} API level itself and 
not just creating the start-time? Or at-least when the actual write happens for 
a given entity?
 - Not sure if we should make these cache-sizes public and document in 
yarn-default.xml. They seem like internal only configs that nobody should touch 
in normal situations. Right?

 Leveldb timeline store needs simple write locking
 -

 Key: YARN-1730
 URL: https://issues.apache.org/jira/browse/YARN-1730
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Billie Rinaldi
Assignee: Billie Rinaldi
 Attachments: YARN-1730.1.patch, YARN-1730.2.patch, YARN-1730.3.patch, 
 YARN-1730.4.patch, YARN-1730.5.patch


 The actual data writes are performed atomically in a batch, but a lock should 
 be held while identifying a start time for the entity, which precedes every 
 write.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493


[ 
https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913743#comment-13913743
 ] 

Jian He commented on YARN-1577:
---

YARN-1389 is adding support for getting attempt report into 
ApplicationClientProtocol, we can take advantage of that to get the attempt 
report.
If people use UnmanagedAMLauncher to launch the unmanaged AM,I think the change 
should be hidden from user.  For the concern of incompatibility, an alternative 
solution would be to hack the app state to return the previous state before 
ACCEPTED state when queried, until the attempt reaches the LAUNCHED state.


 Unmanaged AM is broken because of YARN-1493
 ---

 Key: YARN-1577
 URL: https://issues.apache.org/jira/browse/YARN-1577
 Project: Hadoop YARN
  Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Jian He
Assignee: Naren Koneru
Priority: Blocker

 Today unmanaged AM client is waiting for app state to be Accepted to launch 
 the AM. This is broken since we changed in YARN-1493 to start the attempt 
 after the application is Accepted. We may need to introduce an attempt state 
 report that client can rely on to query the attempt state and choose to 
 launch the unmanaged AM.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493

2014-02-26 Thread Naren Koneru (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913770#comment-13913770
]

Naren Koneru commented on YARN-1577:

Agreed that when people use UnmanagedAMLauncher, the change can be hidden.
However, in some cases, we dont use this class since we dont necessarily spawn
a process for AM in the unmanaged client always. I am ok to break compatibility
for this use case and we can fix the UnmanagedAMLauncher to wait till the
attempt is launched.

As for the hack you mentioned, though that solution seems to be the least
intrusive, I am a little skeptical if this would have any side effects since we
would be lying in the ApplicationClientProtocol. Otherwise, we can get the
changes from YARN-1389 and change the UnmanagedAMLauncher and the llama clients
accordingly.

thoughts??..I am ok with either approach.

Unmanaged AM is broken because of YARN-1493
---

Key: YARN-1577
URL: https://issues.apache.org/jira/browse/YARN-1577
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Jian He
Assignee: Naren Koneru
Priority: Blocker

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate


[ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913790#comment-13913790
 ] 

Hadoop QA commented on YARN-1301:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631387/YARN-1301.6.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3197//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3197//console

This message is automatically generated.

 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-986) YARN should use cluster-id as token service address


 [ 
https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-986:
--

Attachment: yarn-986-2.patch

 YARN should use cluster-id as token service address
 ---

 Key: YARN-986
 URL: https://issues.apache.org/jira/browse/YARN-986
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Karthik Kambatla
Priority: Blocker
 Attachments: yarn-986-1.patch, yarn-986-2.patch, 
 yarn-986-prelim-0.patch


 This needs to be done to support non-ip based fail over of RM. Once the 
 server sets the token service address to be this generic ClusterId/ServiceId, 
 clients can translate it to appropriate final IP and then be able to select 
 tokens via TokenSelectors.
 Some workarounds for other related issues were put in place at YARN-945.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-986) YARN should use cluster-id as token service address

[
https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913822#comment-13913822
]

Karthik Kambatla commented on YARN-986:
---

Thanks for the review, Vinod. Uploaded a patch addressing most of your
comments.

bq. getRMDelegationTokenService() API doesn't belong in this class
Moved it to YarnClient - that seemed the most reasonable place for this.

bq. In the same API, you are only building the service against one address
RM_ADDRESS. That should be qualified by RM-ID?
If you are referring to the non-HA case, we don't need to set RM-ID as it is
used. In the HA case, we do set the RM-ID.

bq. Essentially the new API is what everyone should use to be able to work with
RM fail-over. Why not just deprecate the older API? And change all the callers?
As we discussed offline, the old API is for all other kinds of tokens. IIUC,
only the RMDelegationTokens are affected by the failover. Do you think any
other tokens require augmenting their service names to capture both RM's
service addresses?

bq. We need to be sure distributed-shell and other apps also work fine in this
new world, beyond just MR.
No one else seems to be using RMDelegationTokens, at least in the Hadoop code
base. Apps outside of Hadoop can continue doing what they do to work against
non-HA RM. If they want to run against HA RM, the onus is on them to update.
Should we release note this in 2.4?

YARN should use cluster-id as token service address
---

Key: YARN-986
URL: https://issues.apache.org/jira/browse/YARN-986
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Karthik Kambatla
Priority: Blocker
Attachments: yarn-986-1.patch, yarn-986-2.patch,
yarn-986-prelim-0.patch

This needs to be done to support non-ip based fail over of RM. Once the
server sets the token service address to be this generic ClusterId/ServiceId,
clients can translate it to appropriate final IP and then be able to select
tokens via TokenSelectors.
Some workarounds for other related issues were put in place at YARN-945.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().

[
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913857#comment-13913857
]

Xuan Gong commented on YARN-1410:
-

bq. can we do this in the RM?

Done

bq. Please put some comments in the test to help understand what is being
tested. e.g. testing that failed over RM accepts the appId in submitContext
even though it does not exist internally

added

bq. If the test is doing failover in the test method then why is
submitApplication(ApplicationId oldAppId) also causing failover?

removed

bq. Looks like the RM already blindly accepts the appId present in the
submitContext.

Yes.

bq. Please change the title of the jira and link it to the other 2 jiras.

Done

Handle RM fails over after getApplicationID() and before submitApplication().
-

Key: YARN-1410
URL: https://issues.apache.org/jira/browse/YARN-1410
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
Attachments: YARN-1410-outline.patch, YARN-1410.1.patch,
YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch,
YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch

Original Estimate: 48h
Remaining Estimate: 48h

App submission involves
1) creating appId
2) using that appId to submit an ApplicationSubmissionContext to the user.
The client may have obtained an appId from an RM, the RM may have failed
over, and the client may submit the app to the new RM.
Since the new RM has a different notion of cluster timestamp (used to create
app id) the new RM may reject the app submission resulting in unexpected
failure on the client side.
The same may happen for other 2 step client API operations.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().


 [ 
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-1410:


Attachment: YARN-1410.8.patch

 Handle RM fails over after getApplicationID() and before submitApplication().
 -

 Key: YARN-1410
 URL: https://issues.apache.org/jira/browse/YARN-1410
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, 
 YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, 
 YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 App submission involves
 1) creating appId
 2) using that appId to submit an ApplicationSubmissionContext to the user.
 The client may have obtained an appId from an RM, the RM may have failed 
 over, and the client may submit the app to the new RM.
 Since the new RM has a different notion of cluster timestamp (used to create 
 app id) the new RM may reject the app submission resulting in unexpected 
 failure on the client side.
 The same may happen for other 2 step client API operations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate

2014-02-26 Thread Tsuyoshi OZAWA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913867#comment-13913867
 ] 

Tsuyoshi OZAWA commented on YARN-1301:
--

[~zjshen], thank you for reviewing. Jenkins passed tests :-)

 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy


[ 
https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913884#comment-13913884
 ] 

Vinod Kumar Vavilapalli commented on YARN-1740:
---

Jian, can we specifically test the redirection? The current test seems not much 
useful.

 Redirection from AM-URL is broken with HTTPS_ONLY policy
 

 Key: YARN-1740
 URL: https://issues.apache.org/jira/browse/YARN-1740
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Yesha Vora
Assignee: Jian He
 Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch


 Steps to reproduce:
 1) Run a sleep job
 2) Run: yarn application -list command to find AM URL.
 root@host1:~# yarn application -list
 Total number of applications (application-types: [] and states: SUBMITTED, 
 ACCEPTED, RUNNING):1
 Application-Id Application-Name Application-Type User Queue State Final-State 
 Progress Tracking-URL
 application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING 
 UNDEFINED 5% http://host1:40653
 3) Try to access http://host1:40653/ws/v1/mapreduce/info; url.
 This URL redirects to 
 http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
 Here, Http protocol is used with HTTPS port for RM.
 The expected Url is 
 https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1542) Add unit test for public resource on viewfs


[ 
https://issues.apache.org/jira/browse/YARN-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913898#comment-13913898
 ] 

Vinod Kumar Vavilapalli commented on YARN-1542:
---

Ok.. sure.. I didn't see the blocking ticket earlier. Makes sense. Can you 
address the remaining comments?

 Add unit test for public resource on viewfs
 ---

 Key: YARN-1542
 URL: https://issues.apache.org/jira/browse/YARN-1542
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Reporter: Gera Shegalov
Assignee: Gera Shegalov
 Attachments: YARN-1542.v01.patch, YARN-1542.v02.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


 [ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-1429:
---

Summary: *nix: Allow a way for users to augment classpath of YARN daemons  
(was: YARN_CLASSPATH is referenced in yarn command comments but doesn't do 
anything)

 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


[ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913903#comment-13913903
 ] 

Karthik Kambatla commented on YARN-1429:


Verified the linux version of the patch. Couldn't get the yarn.cmd changes to 
apply cleanly - and don't have access to a Windows machine. Created YARN-1767 
for the Windows version. 

+1. Committing this shortly. 

 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1363) Get / Cancel / Renew delegation token api should be non blocking


[ 
https://issues.apache.org/jira/browse/YARN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913910#comment-13913910
 ] 

Hadoop QA commented on YARN-1363:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631388/YARN-1363.10.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 11 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  org.apache.hadoop.mapreduce.v2.TestUberAM
  org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

  The following test timeouts occurred in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3198//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3198//console

This message is automatically generated.

 Get / Cancel / Renew delegation token api should be non blocking
 

 Key: YARN-1363
 URL: https://issues.apache.org/jira/browse/YARN-1363
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Omkar Vinit Joshi
Assignee: Zhijie Shen
 Attachments: YARN-1363.1.patch, YARN-1363.10.patch, 
 YARN-1363.2.patch, YARN-1363.3.patch, YARN-1363.4.patch, YARN-1363.5.patch, 
 YARN-1363.6.patch, YARN-1363.7.patch, YARN-1363.8.patch, YARN-1363.9.patch


 Today GetDelgationToken, CancelDelegationToken and RenewDelegationToken are 
 all blocking apis.
 * As a part of these calls we try to update RMStateStore and that may slow it 
 down.
 * Now as we have limited number of client request handlers we may fill up 
 client handlers quickly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


 [ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-1429:
---

Hadoop Flags: Reviewed

 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Fix For: 2.5.0

 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


[ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913944#comment-13913944
 ] 

Karthik Kambatla commented on YARN-1429:


Thanks Jarcec for the contribution. Just committed this to trunk and branch-2. 

 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Fix For: 2.5.0

 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


[ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913961#comment-13913961
 ] 

Karthik Kambatla commented on YARN-1429:


Forgot to mention. Thought about a better username, but the approach in the 
patch of using YARN_USER_CLASSPATH and updating the description in comments to 
capture the fact that it augments seemed enough to me. 

 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Fix For: 2.5.0

 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1429) *nix: Allow a way for users to augment classpath of YARN daemons


[ 
https://issues.apache.org/jira/browse/YARN-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914007#comment-13914007
 ] 

Hudson commented on YARN-1429:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5236 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5236/])
YARN-1429. *nix: Allow a way for users to augment classpath of YARN daemons. 
(Jarek Jarcec Cecho via kasha) (kasha: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1572405)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn


 *nix: Allow a way for users to augment classpath of YARN daemons
 

 Key: YARN-1429
 URL: https://issues.apache.org/jira/browse/YARN-1429
 Project: Hadoop YARN
  Issue Type: Bug
  Components: client
Reporter: Sandy Ryza
Assignee: Jarek Jarcec Cecho
Priority: Trivial
  Labels: newbie
 Fix For: 2.5.0

 Attachments: YARN-1429.linux.patch, YARN-1429.patch, YARN-1429.patch


 YARN_CLASSPATH is referenced in the comments in 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn and 
 ./hadoop-yarn-project/hadoop-yarn/bin/yarn.cmd, but doesn't do anything.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1301) Need to log the blacklist additions/removals when YarnSchedule#allocate


[ 
https://issues.apache.org/jira/browse/YARN-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914008#comment-13914008
 ] 

Hudson commented on YARN-1301:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #5236 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/5236/])
YARN-1301. Added the INFO level log of the non-empty blacklist additions and 
removals inside ApplicationMasterService. Contributed by Tsuyoshi Ozawa. 
(zjshen: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1572400)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java


 Need to log the blacklist additions/removals when YarnSchedule#allocate
 ---

 Key: YARN-1301
 URL: https://issues.apache.org/jira/browse/YARN-1301
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.4.0

 Attachments: YARN-1301.1.patch, YARN-1301.2.patch, YARN-1301.3.patch, 
 YARN-1301.4.patch, YARN-1301.5.patch, YARN-1301.6.patch


 Now without the log, it's hard to debug whether blacklist is updated on the 
 scheduler side or not



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (YARN-1754) Container process is not really killed

[
https://issues.apache.org/jira/browse/YARN-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli resolved YARN-1754.
---

Resolution: Duplicate

May be if setsid is not available, we can try traversing the process-tree and
kill those processes. It is still fraught with race conditions but useful
nonetheless.

IAC, this is a known issue, closing as duplicate of YARN-76.

Container process is not really killed
--

Key: YARN-1754
URL: https://issues.apache.org/jira/browse/YARN-1754
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.2.0
Environment: Mac
Reporter: Jeff Zhang

I test the following distributed shell example on my mac:
hadoop jar
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar
-appname shell -jar
share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar
-shell_command=sleep -shell_args=10 -num_containers=1
And it will start 2 process for one container, one is the shell process,
another is the real command I execute ( here is sleep 10).
And then I kill this application by running command yarn application -kill
app_id
it will kill the shell process, but won't kill the real command process. The
reason is that yarn use kill command to kill process, but it won't kill its
child process. use pkill could resolve this issue.
IMHO, it is a very important case which will make the resource usage
inconsistency, and have potential security problem.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1701) More intuitive defaults for AHS

[
https://issues.apache.org/jira/browse/YARN-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914022#comment-13914022
]

Vinod Kumar Vavilapalli commented on YARN-1701:
---

I agree the configuration needs to be simple. So let's definitely change the
store. Specially given we also have a flag to enable/disable it.

Re the path, I am going to side with [~jira.shegalov] on this one. I have
already seen a couple of users running into the same issue of the unusual
default path when put on HDFS.

I actually think we should change all other defaults which point to
hadoop.log.dir. hadoop.log.dir clearly hints to a local file-system path, in
all of Hadoop. We have historically used /tmp paths as default in Hadoop.

More intuitive defaults for AHS
---

Key: YARN-1701
URL: https://issues.apache.org/jira/browse/YARN-1701
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.4.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: YARN-1701.v01.patch

When I enable AHS via yarn.ahs.enabled, the app history is still not visible
in AHS webUI. This is due to NullApplicationHistoryStore as
yarn.resourcemanager.history-writer.class. It would be good to have just one
key to enable basic functionality.
yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is
local file system location. However, FileSystemApplicationHistoryStore uses
DFS by default.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1410) Handle RM fails over after getApplicationID() and before submitApplication().


[ 
https://issues.apache.org/jira/browse/YARN-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914036#comment-13914036
 ] 

Hadoop QA commented on YARN-1410:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12631415/YARN-1410.8.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/3200//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3200//console

This message is automatically generated.

 Handle RM fails over after getApplicationID() and before submitApplication().
 -

 Key: YARN-1410
 URL: https://issues.apache.org/jira/browse/YARN-1410
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Xuan Gong
 Attachments: YARN-1410-outline.patch, YARN-1410.1.patch, 
 YARN-1410.2.patch, YARN-1410.2.patch, YARN-1410.3.patch, YARN-1410.4.patch, 
 YARN-1410.5.patch, YARN-1410.6.patch, YARN-1410.7.patch, YARN-1410.8.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 App submission involves
 1) creating appId
 2) using that appId to submit an ApplicationSubmissionContext to the user.
 The client may have obtained an appId from an RM, the RM may have failed 
 over, and the client may submit the app to the new RM.
 Since the new RM has a different notion of cluster timestamp (used to create 
 app id) the new RM may reject the app submission resulting in unexpected 
 failure on the client side.
 The same may happen for other 2 step client API operations.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1599) webUI rm.webapp.AppBlock should redirect to a history App page if and when available

[
https://issues.apache.org/jira/browse/YARN-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914042#comment-13914042
]

Vinod Kumar Vavilapalli commented on YARN-1599:
---

I actually argue the contrary. We still have many cases where we don't have
tracking URLs for whatever reasons. So, I proposed a solution quite opposite to
what is being put up here - to always force the users to land on the RM page -
see YARN-1140. It is more clicks as argued here, but at the least we can then
be consistent. Similarly see YARN-1106.

I can't find the JIRA but given bulk of YARN-321 is done, we are changing the
RM UI to add much more information than is present. Clearly the per-framework
UI RM UI in terms of data richness but the second UI isn't trivial anymore.

bq. our users think that the AppMaster logs were lost because the link to the
AM attempt logs are not updated and result in HTTP 404
Why don't we fix this issue if that is the driving force. I don't expect a 404
if users go through the proxy (which they should!). Can you throw more light on
when this can happen?

webUI rm.webapp.AppBlock should redirect to a history App page if and when
available

Key: YARN-1599
URL: https://issues.apache.org/jira/browse/YARN-1599
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.0.5-alpha, 2.2.0
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: Screen Shot 2014-01-16 at 6.52.17 PM.png, Screen Shot
2014-01-16 at 7.30.32 PM.png, YARN-1599.v01.patch, YARN-1599.v02.patch,
YARN-1599.v03.patch

When the log aggregation is enabled, and the application finishes, our users
think that the AppMaster logs were lost because the link to the AM attempt
logs are not updated and result in HTTP 404. Only tracking URL is updated. In
order to have a smoother user experience, we propose to simply redirect to
the new tracking URL when the page with invalid log links is accessed.

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1577) Unmanaged AM is broken because of YARN-1493

[
https://issues.apache.org/jira/browse/YARN-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914090#comment-13914090
]

Vinod Kumar Vavilapalli commented on YARN-1577:
---

Let's not put in any hacks. Using ApplicationAttemptReport is the right
approach.

Unmanaged AMs are supposed to be used by invoking the Unmanaged-AM tool itself.
We never made the underlying state interaction in the ResourceManager a public
interface.

bq. However, this would not be backwards compatible since it involves changes
to the unmanaged clients. Since I do not see any documentation for the
unmanaged clients, is this acceptable?
It isn't an incompatible change. Like I said above, the AM-RM life-cycle isn't
a publicly supported interface if not used via the unmanaged-AM tool.

Clearly we need documentation stating these things.

Unmanaged AM is broken because of YARN-1493
---

Key: YARN-1577
URL: https://issues.apache.org/jira/browse/YARN-1577
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.3.0
Reporter: Jian He
Assignee: Naren Koneru
Priority: Blocker

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1752) Unexpected Unregistered event at Attempt Launched state

2014-02-26 Thread Rohith (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914134#comment-13914134
 ] 

Rohith commented on YARN-1752:
--

I agree that fix should be from YARN. I was thinking only from 
MapReduce(MRAppMaster) perspective so this confusion happened :-( I will create 
jira under MapReduce project.


 Unexpected Unregistered event at Attempt Launched state
 ---

 Key: YARN-1752
 URL: https://issues.apache.org/jira/browse/YARN-1752
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
 Attachments: YARN-1752.1.patch


 {code}
 2014-02-21 14:56:03,453 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
 Can't handle this event at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 UNREGISTERED at LAUNCHED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
   at java.lang.Thread.run(Thread.java:695)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1752) Unexpected Unregistered event at Attempt Launched state

2014-02-26 Thread Rohith (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith updated YARN-1752:
-

Attachment: YARN-1752.2.patch

Attaching patch for YARN fix.

The idea behind patch is checking for responseID(to -1) for an attempt. 
Basically, when an app attempt is registered to ApplicationMasterService, 
AllocateResponse Id  is set to -1. This response id set to zero only if 
ApplicationMaster is registered to an attempt.

 Unexpected Unregistered event at Attempt Launched state
 ---

 Key: YARN-1752
 URL: https://issues.apache.org/jira/browse/YARN-1752
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Rohith
 Attachments: YARN-1752.1.patch, YARN-1752.2.patch


 {code}
 2014-02-21 14:56:03,453 ERROR 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
 Can't handle this event at current state
 org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: 
 UNREGISTERED at LAUNCHED
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
   at 
 org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
   at 
 org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
   at java.lang.Thread.run(Thread.java:695)
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (YARN-1740) Redirection from AM-URL is broken with HTTPS_ONLY policy


 [ 
https://issues.apache.org/jira/browse/YARN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-1740:
--

Attachment: YARN-1740.4.patch

Changed the test case to test the AM url redirected properly in both http and 
https scenario.

 Redirection from AM-URL is broken with HTTPS_ONLY policy
 

 Key: YARN-1740
 URL: https://issues.apache.org/jira/browse/YARN-1740
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Yesha Vora
Assignee: Jian He
 Attachments: YARN-1740.1.patch, YARN-1740.2.patch, YARN-1740.3.patch, 
 YARN-1740.4.patch


 Steps to reproduce:
 1) Run a sleep job
 2) Run: yarn application -list command to find AM URL.
 root@host1:~# yarn application -list
 Total number of applications (application-types: [] and states: SUBMITTED, 
 ACCEPTED, RUNNING):1
 Application-Id Application-Name Application-Type User Queue State Final-State 
 Progress Tracking-URL
 application_1383251398986_0003 Sleep job MAPREDUCE hdfs default RUNNING 
 UNDEFINED 5% http://host1:40653
 3) Try to access http://host1:40653/ws/v1/mapreduce/info; url.
 This URL redirects to 
 http://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info
 Here, Http protocol is used with HTTPS port for RM.
 The expected Url is 
 https://RM_host:RM_https_port/proxy/application_1383251398986_0003/ws/v1/mapreduce/info



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1713) Implement getnewapplication and submitapp as part of RM web service


[ 
https://issues.apache.org/jira/browse/YARN-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13914166#comment-13914166
 ] 

Jian He commented on YARN-1713:
---

- The following piece can be incorporated into the above 
ApplicationSubmissionContext.newInstance(
{code}
appContext.setKeepContainersAcrossApplicationAttempts(
  newApp.isKeepContainers());
{code}

- hasQueueAccess is not used anywhere? I think we can just use hasAccess() in 
all the places and parameterize ApplicationAccessType and ApplicationAccessType?
- Rename AppSubmissionInfo to AppSubmissionContextInfo ?, similarly 
name=appsubmission - “appsubmissioncontext”. 
 Similarly, ContainerLaunchInfo - ContainerLaunchContextInfo
- AppSubmissionInfo: we can initialize cancelTokensWhenComplete to true, 
keepContainers to false

 Implement getnewapplication and submitapp as part of RM web service
 ---

 Key: YARN-1713
 URL: https://issues.apache.org/jira/browse/YARN-1713
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-1713.3.patch, apache-yarn-1713.4.patch, 
 apache-yarn-1713.cumulative.2.patch, apache-yarn-1713.cumulative.3.patch, 
 apache-yarn-1713.cumulative.patch, apache-yarn-1713.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (YARN-1752) Unexpected Unregistered event at Attempt Launched state