date:20130925


[ 
https://issues.apache.org/jira/browse/YARN-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777345#comment-13777345
 ] 

Hudson commented on YARN-1229:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #343 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/343/])
YARN-1229. Define constraints on Auxiliary Service names. Change ShuffleHandler 
service name from mapreduce.shuffle to mapreduce_shuffle. Contributed by Xuan 
Gong. (sseth: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526065)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/SingleCluster.apt.vm
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java


 Shell$ExitCodeException could happen if AM fails to start
 -

 Key: YARN-1229
 URL: https://issues.apache.org/jira/browse/YARN-1229
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.1.1-beta
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Priority: Blocker
 Fix For: 2.1.2-beta

 Attachments: YARN-1229.1.patch, YARN-1229.2.patch, YARN-1229.3.patch, 
 YARN-1229.4.patch, YARN-1229.5.patch, YARN-1229.6.patch


 I run sleep job. If AM fails to start, this exception could occur:
 13/09/20 11:00:23 INFO mapreduce.Job: Job job_1379673267098_0020 failed with 
 state FAILED due to: Application application_1379673267098_0020 failed 1 
 times due to AM Container for appattempt_1379673267098_0020_01 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: 
 /myappcache/application_1379673267098_0020/container_1379673267098_0020_01_01/launch_container.sh:
  line 12: export: 
 `NM_AUX_SERVICE_mapreduce.shuffle=AAA0+gA=
 ': not a valid identifier
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:270)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:78)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 .Failing this attempt.. Failing the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1204) Need to add https port related property in Yarn


[ 
https://issues.apache.org/jira/browse/YARN-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777347#comment-13777347
 ] 

Hudson commented on YARN-1204:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #343 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/343/])
YARN-1204. Added separate configuration properties for https for RM and NM 
without which servers enabled with https will also start on http ports. 
Contributed by Omkar Vinit Joshi.
MAPREDUCE-5523. Added separate configuration properties for https for JHS 
without which even when https is enabled, it starts on http port itself. 
Contributed by Omkar Vinit Joshi. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1525947)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/WebAppUtil.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmFilterInitializer.java


 Need to add https port related property in Yarn
 ---

 Key: YARN-1204
 URL: https://issues.apache.org/jira/browse/YARN-1204
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
 Fix For: 2.1.2-beta

 Attachments: YARN-1204.20131018.1.patch, YARN-1204.20131020.1.patch, 
 YARN-1204.20131020.2.patch, YARN-1204.20131020.3.patch, 
 YARN-1204.20131020.4.patch, YARN-1204.20131023.1.patch


 There is no yarn property available to configure https port for Resource 
 manager, nodemanager and history server. Currently, Yarn services uses the 
 port defined for http [defined by 
 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', 
 'yarn.resourcemanager.webapp.address'] for running services on https protocol.
 Yarn should have list of property to assign https port for RM, NM and JHS.
 It can be like below.
 yarn.nodemanager.webapp.https.address
 yarn.resourcemanager.webapp.https.address
 mapreduce.jobhistory.webapp.https.address

[jira] [Commented] (YARN-1214) Register ClientToken MasterKey in SecretManager after it is saved


[ 
https://issues.apache.org/jira/browse/YARN-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777346#comment-13777346
 ] 

Hudson commented on YARN-1214:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #343 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/343/])
YARN-1214. Register ClientToken MasterKey in SecretManager after it is saved 
(Jian He via bikas) (bikas: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526078)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java


 Register ClientToken MasterKey in SecretManager after it is saved
 -

 Key: YARN-1214
 URL: https://issues.apache.org/jira/browse/YARN-1214
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Priority: Critical
 Fix For: 2.1.2-beta

 Attachments: YARN-1214.1.patch, YARN-1214.2.patch, YARN-1214.3.patch, 
 YARN-1214.4.patch, YARN-1214.5.patch, YARN-1214.6.patch, YARN-1214.patch


 Currently, app attempt ClientToken master key is registered before it is 
 saved. This can cause problem that before the master key is saved, client 
 gets the token and RM also crashes, RM cannot reloads the master key back 
 after it restarts as it is not saved. As a result, client is holding an 
 invalid token.
 We can register the client token master key after it is saved in the store.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1229) Shell$ExitCodeException could happen if AM fails to start


[ 
https://issues.apache.org/jira/browse/YARN-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777442#comment-13777442
 ] 

Hudson commented on YARN-1229:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1559 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1559/])
YARN-1229. Define constraints on Auxiliary Service names. Change ShuffleHandler 
service name from mapreduce.shuffle to mapreduce_shuffle. Contributed by Xuan 
Gong. (sseth: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526065)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/SingleCluster.apt.vm
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java


 Shell$ExitCodeException could happen if AM fails to start
 -

 Key: YARN-1229
 URL: https://issues.apache.org/jira/browse/YARN-1229
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.1.1-beta
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Priority: Blocker
 Fix For: 2.1.2-beta

 Attachments: YARN-1229.1.patch, YARN-1229.2.patch, YARN-1229.3.patch, 
 YARN-1229.4.patch, YARN-1229.5.patch, YARN-1229.6.patch


 I run sleep job. If AM fails to start, this exception could occur:
 13/09/20 11:00:23 INFO mapreduce.Job: Job job_1379673267098_0020 failed with 
 state FAILED due to: Application application_1379673267098_0020 failed 1 
 times due to AM Container for appattempt_1379673267098_0020_01 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: 
 /myappcache/application_1379673267098_0020/container_1379673267098_0020_01_01/launch_container.sh:
  line 12: export: 
 `NM_AUX_SERVICE_mapreduce.shuffle=AAA0+gA=
 ': not a valid identifier
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:270)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:78)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 .Failing this attempt.. Failing the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1204) Need to add https port related property in Yarn


[ 
https://issues.apache.org/jira/browse/YARN-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777444#comment-13777444
 ] 

Hudson commented on YARN-1204:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1559 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1559/])
YARN-1204. Added separate configuration properties for https for RM and NM 
without which servers enabled with https will also start on http ports. 
Contributed by Omkar Vinit Joshi.
MAPREDUCE-5523. Added separate configuration properties for https for JHS 
without which even when https is enabled, it starts on http port itself. 
Contributed by Omkar Vinit Joshi. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1525947)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/WebAppUtil.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmFilterInitializer.java


 Need to add https port related property in Yarn
 ---

 Key: YARN-1204
 URL: https://issues.apache.org/jira/browse/YARN-1204
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
 Fix For: 2.1.2-beta

 Attachments: YARN-1204.20131018.1.patch, YARN-1204.20131020.1.patch, 
 YARN-1204.20131020.2.patch, YARN-1204.20131020.3.patch, 
 YARN-1204.20131020.4.patch, YARN-1204.20131023.1.patch


 There is no yarn property available to configure https port for Resource 
 manager, nodemanager and history server. Currently, Yarn services uses the 
 port defined for http [defined by 
 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', 
 'yarn.resourcemanager.webapp.address'] for running services on https protocol.
 Yarn should have list of property to assign https port for RM, NM and JHS.
 It can be like below.
 yarn.nodemanager.webapp.https.address
 yarn.resourcemanager.webapp.https.address

[jira] [Commented] (YARN-1214) Register ClientToken MasterKey in SecretManager after it is saved


[ 
https://issues.apache.org/jira/browse/YARN-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777455#comment-13777455
 ] 

Hudson commented on YARN-1214:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1533 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1533/])
YARN-1214. Register ClientToken MasterKey in SecretManager after it is saved 
(Jian He via bikas) (bikas: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526078)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ClientToAMTokenSecretManagerInRM.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestRMStateStore.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java


 Register ClientToken MasterKey in SecretManager after it is saved
 -

 Key: YARN-1214
 URL: https://issues.apache.org/jira/browse/YARN-1214
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Jian He
Assignee: Jian He
Priority: Critical
 Fix For: 2.1.2-beta

 Attachments: YARN-1214.1.patch, YARN-1214.2.patch, YARN-1214.3.patch, 
 YARN-1214.4.patch, YARN-1214.5.patch, YARN-1214.6.patch, YARN-1214.patch


 Currently, app attempt ClientToken master key is registered before it is 
 saved. This can cause problem that before the master key is saved, client 
 gets the token and RM also crashes, RM cannot reloads the master key back 
 after it restarts as it is not saved. As a result, client is holding an 
 invalid token.
 We can register the client token master key after it is saved in the store.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1204) Need to add https port related property in Yarn


[ 
https://issues.apache.org/jira/browse/YARN-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777457#comment-13777457
 ] 

Hudson commented on YARN-1204:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1533 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1533/])
YARN-1204. Added separate configuration properties for https for RM and NM 
without which servers enabled with https will also start on http ports. 
Contributed by Omkar Vinit Joshi.
MAPREDUCE-5523. Added separate configuration properties for https for JHS 
without which even when https is enabled, it starts on http port itself. 
Contributed by Omkar Vinit Joshi. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1525947)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/WebAppUtil.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxy.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmFilterInitializer.java


 Need to add https port related property in Yarn
 ---

 Key: YARN-1204
 URL: https://issues.apache.org/jira/browse/YARN-1204
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
 Fix For: 2.1.2-beta

 Attachments: YARN-1204.20131018.1.patch, YARN-1204.20131020.1.patch, 
 YARN-1204.20131020.2.patch, YARN-1204.20131020.3.patch, 
 YARN-1204.20131020.4.patch, YARN-1204.20131023.1.patch


 There is no yarn property available to configure https port for Resource 
 manager, nodemanager and history server. Currently, Yarn services uses the 
 port defined for http [defined by 
 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', 
 'yarn.resourcemanager.webapp.address'] for running services on https protocol.
 Yarn should have list of property to assign https port for RM, NM and JHS.
 It can be like below.
 yarn.nodemanager.webapp.https.address
 yarn.resourcemanager.webapp.https.address
 mapreduce.jobhistory.webapp.https.address

[jira] [Commented] (YARN-1229) Shell$ExitCodeException could happen if AM fails to start


[ 
https://issues.apache.org/jira/browse/YARN-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777454#comment-13777454
 ] 

Hudson commented on YARN-1229:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1533 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1533/])
YARN-1229. Define constraints on Auxiliary Service names. Change ShuffleHandler 
service name from mapreduce.shuffle to mapreduce_shuffle. Contributed by Xuan 
Gong. (sseth: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526065)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/SingleCluster.apt.vm
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java


 Shell$ExitCodeException could happen if AM fails to start
 -

 Key: YARN-1229
 URL: https://issues.apache.org/jira/browse/YARN-1229
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.1.1-beta
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Priority: Blocker
 Fix For: 2.1.2-beta

 Attachments: YARN-1229.1.patch, YARN-1229.2.patch, YARN-1229.3.patch, 
 YARN-1229.4.patch, YARN-1229.5.patch, YARN-1229.6.patch


 I run sleep job. If AM fails to start, this exception could occur:
 13/09/20 11:00:23 INFO mapreduce.Job: Job job_1379673267098_0020 failed with 
 state FAILED due to: Application application_1379673267098_0020 failed 1 
 times due to AM Container for appattempt_1379673267098_0020_01 exited 
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: 
 /myappcache/application_1379673267098_0020/container_1379673267098_0020_01_01/launch_container.sh:
  line 12: export: 
 `NM_AUX_SERVICE_mapreduce.shuffle=AAA0+gA=
 ': not a valid identifier
 at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
 at org.apache.hadoop.util.Shell.run(Shell.java:379)
 at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
 at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:270)
 at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:78)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 .Failing this attempt.. Failing the application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-986) YARN should have a ClusterId/ServiceId

2013-09-25 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777523#comment-13777523
 ] 

Daryn Sharp commented on YARN-986:
--

This sounds like NN HA tokens which IMHO are rather hacky.  I've been intending 
to take advantage of my RPCv9 auth changes for the server to tell the client 
the token service (or perhaps another field) it needs to decouple tokens 
entirely from IP/hostname.  Thoughts on this approach?

 YARN should have a ClusterId/ServiceId
 --

 Key: YARN-986
 URL: https://issues.apache.org/jira/browse/YARN-986
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Karthik Kambatla

 This needs to be done to support non-ip based fail over of RM. Once the 
 server sets the token service address to be this generic ClusterId/ServiceId, 
 clients can translate it to appropriate final IP and then be able to select 
 tokens via TokenSelectors.
 Some workarounds for other related issues were put in place at YARN-945.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-90) NodeManager should identify failed disks becoming good back again

2013-09-25 Thread nijel (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nijel updated YARN-90:
--

Attachment: YARN-90.1.patch

 NodeManager should identify failed disks becoming good back again
 -

 Key: YARN-90
 URL: https://issues.apache.org/jira/browse/YARN-90
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: nodemanager
Reporter: Ravi Gummadi
 Attachments: YARN-90.1.patch, YARN-90.patch


 MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes 
 down, it is marked as failed forever. To reuse that disk (after it becomes 
 good), NodeManager needs restart. This JIRA is to improve NodeManager to 
 reuse good disks(which could be bad some time back).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777572#comment-13777572
]

Sandy Ryza commented on YARN-1228:
--

Updated patch includes a test

Clean up Fair Scheduler configuration loading
-

Key: YARN-1228
URL: https://issues.apache.org/jira/browse/YARN-1228
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.1.1-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-1228-1.patch, YARN-1228.patch

Currently the Fair Scheduler is configured in two ways
* An allocations file that has a different format than the standard Hadoop
configuration file, which makes it easier to specify hierarchical objects
like queues and their properties.
* With properties like yarn.scheduler.fair.max.assign that are specified in
the standard Hadoop configuration format.
The standard and default way of configuring it is to use fair-scheduler.xml
as the allocations file and to put the yarn.scheduler properties in
yarn-site.xml.
It is also possible to specify a different file as the allocations file, and
to place the yarn.scheduler properties in fair-scheduler.xml, which will be
interpreted as in the standard Hadoop configuration format. This flexibility
is both confusing and unnecessary.
Additionally, the allocation file is loaded as fair-scheduler.xml from the
classpath if it is not specified, but is loaded as a File if it is. This
causes two problems
1. We see different behavior when not setting the
yarn.scheduler.fair.allocation.file, and setting it to fair-scheduler.xml,
which is its default.
2. Classloaders may choose to cache resources, which can break the reload
logic when yarn.scheduler.fair.allocation.file is not specified.
We should never allow the yarn.scheduler properties to go into
fair-scheduler.xml. And we should always load the allocations file as a
file, not as a resource on the classpath. To preserve existing behavior and
allow loading files from the classpath, we can look for files on the
classpath, but strip of their scheme and interpret them as Files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sandy Ryza updated YARN-1228:
-

Attachment: YARN-1228-1.patch

Clean up Fair Scheduler configuration loading
-

[jira] [Commented] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777599#comment-13777599
]

Hadoop QA commented on YARN-1228:
-

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605018/YARN-1228-1.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.

{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.

{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2013//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2013//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2013//console

This message is automatically generated.

Clean up Fair Scheduler configuration loading
-

[jira] [Created] (YARN-1236) FairScheduler setting queue name in RMApp is not working

Sandy Ryza created YARN-1236:


 Summary: FairScheduler setting queue name in RMApp is not working 
 Key: YARN-1236
 URL: https://issues.apache.org/jira/browse/YARN-1236
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-1236.patch

The fair scheduler sometimes picks a different queue than the one an 
application was submitted to, such as when user-as-default-queue is turned on.  
It needs to update the queue name in the RMApp so that this choice will be 
reflected in the UI.

This isn't working because the scheduler is looking up the RMApp by application 
attempt id instead of app id and failing to find it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777661#comment-13777661
]

Sandy Ryza commented on YARN-1228:
--

Updated patch adds license header to test-fair-scheduler.xml

Clean up Fair Scheduler configuration loading
-

[jira] [Updated] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sandy Ryza updated YARN-1228:
-

Attachment: YARN-1228-2.patch

Clean up Fair Scheduler configuration loading
-

[jira] [Commented] (YARN-1236) FairScheduler setting queue name in RMApp is not working


[ 
https://issues.apache.org/jira/browse/YARN-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777694#comment-13777694
 ] 

Hadoop QA commented on YARN-1236:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12605023/YARN-1236.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2014//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2014//console

This message is automatically generated.

 FairScheduler setting queue name in RMApp is not working 
 -

 Key: YARN-1236
 URL: https://issues.apache.org/jira/browse/YARN-1236
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-1236.patch


 The fair scheduler sometimes picks a different queue than the one an 
 application was submitted to, such as when user-as-default-queue is turned 
 on.  It needs to update the queue name in the RMApp so that this choice will 
 be reflected in the UI.
 This isn't working because the scheduler is looking up the RMApp by 
 application attempt id instead of app id and failing to find it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1228) Clean up Fair Scheduler configuration loading

[
https://issues.apache.org/jira/browse/YARN-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1302#comment-1302
]

Hadoop QA commented on YARN-1228:
-

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605025/YARN-1228-2.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2015//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2015//console

This message is automatically generated.

Clean up Fair Scheduler configuration loading
-

[jira] [Commented] (YARN-1232) Configuration support for RM HA


[ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1334#comment-1334
 ] 

Hadoop QA commented on YARN-1232:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12605030/yarn-1232-3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2016//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2016//console

This message is automatically generated.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-986) YARN should have a ClusterId/ServiceId

[
https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1396#comment-1396
]

Bikas Saha commented on YARN-986:
-

This jira has been repurposed beyond tokens. We need a logical name for the RM
instances so that they can correctly pick their configs and clients can
identify them by name. Something like the namespace id in HDFS.

By the time we come to security and tokens, it would make sense to use the
approach that makes configuring it easy for users.
Can you please provide details of what you were thinking. Our initial idea was
to have the RM instance identify its token with its own logical id. Then the
clients will translate the logical id to the physical id of the instance based
on their configuration.

YARN should have a ClusterId/ServiceId
--

Key: YARN-986
URL: https://issues.apache.org/jira/browse/YARN-986
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Karthik Kambatla

This needs to be done to support non-ip based fail over of RM. Once the
server sets the token service address to be this generic ClusterId/ServiceId,
clients can translate it to appropriate final IP and then be able to select
tokens via TokenSelectors.
Some workarounds for other related issues were put in place at YARN-945.

[jira] [Commented] (YARN-899) Get queue administration ACLs working

2013-09-25 Thread Zhijie Shen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1399#comment-1399
 ] 

Zhijie Shen commented on YARN-899:
--

1. Is the following code necessary?
{code}
+System.out.println(callerUGI);
{code}

2. Why not defining checkAccess in YarnScheduler? I saw other ACL related 
method is defined there as well.
{code}
+  boolean checkAccess(UserGroupInformation callerUGI,
+  QueueACL acl, String queueName);
{code}

3. checkViewAccess is used in getApplications and getApplicationReport, does 
|| mean the user doesn't need to have the access to the application whenever 
it has the access to the queue?
{code}
-  private boolean checkAccess(UserGroupInformation callerUGI, String owner,
-  ApplicationAccessType operationPerformed, ApplicationId applicationId) {
+  private boolean checkViewAccess(UserGroupInformation callerUGI, String owner,
+  ApplicationAccessType operationPerformed,
+  RMApp application) {
 return applicationsACLsManager.checkAccess(callerUGI, operationPerformed,
-owner, applicationId);
+owner, application.getApplicationId())
+|| queueACLsManager.checkAccess(callerUGI, QueueACL.ADMINISTER_QUEUE,
+application.getQueue());
{code}

 Get queue administration ACLs working
 -

 Key: YARN-899
 URL: https://issues.apache.org/jira/browse/YARN-899
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Xuan Gong
 Attachments: YARN-899.1.patch, YARN-899.2.patch, YARN-899.3.patch, 
 YARN-899.4.patch, YARN-899.5.patch, YARN-899.5.patch, YARN-899.6.patch, 
 YARN-899.7.patch, YARN-899.8.patch


 The Capacity Scheduler documents the 
 yarn.scheduler.capacity.root.queue-path.acl_administer_queue config option 
 for controlling who can administer a queue, but it is not hooked up to 
 anything.  The Fair Scheduler could make use of a similar option as well.  
 This is a feature-parity regression from MR1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-986) YARN should have a ClusterId/ServiceId


[ 
https://issues.apache.org/jira/browse/YARN-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777819#comment-13777819
 ] 

Karthik Kambatla commented on YARN-986:
---

[~bikassaha], along with other changes, I added {{HAUtil#getLogicalName()}} in 
YARN-1232. Do you think we can just use the same?

 YARN should have a ClusterId/ServiceId
 --

 Key: YARN-986
 URL: https://issues.apache.org/jira/browse/YARN-986
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Karthik Kambatla

 This needs to be done to support non-ip based fail over of RM. Once the 
 server sets the token service address to be this generic ClusterId/ServiceId, 
 clients can translate it to appropriate final IP and then be able to select 
 tokens via TokenSelectors.
 Some workarounds for other related issues were put in place at YARN-945.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1232) Configuration support for RM HA


[ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777832#comment-13777832
 ] 

Bikas Saha commented on YARN-1232:
--

A description of overall approach will help understand this better.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1232) Configuration support for RM HA


[ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777839#comment-13777839
 ] 

Bikas Saha commented on YARN-1232:
--

Secondly, it would be very helpful for existing configs to work in non-HA 
clusters.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1068) Add admin support for HA operations


[ 
https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777849#comment-13777849
 ] 

Bikas Saha commented on YARN-1068:
--

Lets see how YARN-986 and YARN-1232 develop and make a call based on that. 
Would like to avoid committing stuff that will soon be changed a lot.

 Add admin support for HA operations
 ---

 Key: YARN-1068
 URL: https://issues.apache.org/jira/browse/YARN-1068
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Affects Versions: 2.1.0-beta
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1068-1.patch, yarn-1068-2.patch, yarn-1068-3.patch, 
 yarn-1068-4.patch, yarn-1068-5.patch, yarn-1068-6.patch, yarn-1068-7.patch, 
 yarn-1068-prelim.patch


 Support HA admin operations to facilitate transitioning the RM to Active and 
 Standby states.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1237) Description for yarn.nodemanager.aux-services in yarn-default.xml is misleading

2013-09-25 Thread Hitesh Shah (JIRA)

Hitesh Shah created YARN-1237:
-

 Summary: Description for yarn.nodemanager.aux-services in 
yarn-default.xml is misleading
 Key: YARN-1237
 URL: https://issues.apache.org/jira/browse/YARN-1237
 Project: Hadoop YARN
  Issue Type: Bug
  Components: documentation
Reporter: Hitesh Shah
Priority: Minor


Description states:

the valid service name should only contain a-zA-Z0-9_ and can not start with 
numbers 

It seems to indicate only one service is supported. If multiple services are 
allowed, it does not indicate how they should be specified i.e. comma-separated 
or space-separated? If the service name cannot contain spaces, does this imply 
that space-separated lists are also permitted?
 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1238) Fix use of Shuffle service after changes in YARN/MR due to YARN-1229

2013-09-25 Thread Hitesh Shah (JIRA)

Hitesh Shah created YARN-1238:
-

 Summary: Fix use of Shuffle service after changes in YARN/MR due 
to YARN-1229
 Key: YARN-1238
 URL: https://issues.apache.org/jira/browse/YARN-1238
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah
Assignee: Hitesh Shah


YARN-1229 changed the name of the shuffle service.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1089) Add YARN compute units alongside virtual cores

2013-09-25 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777928#comment-13777928
 ] 

Steve Loughran commented on YARN-1089:
--

My stance on the viability of any arbitrary compute unit is known, I'm not 
going to go there in this JIRA.

The patch must not break all code that creates a new Resource. Not only to 
avoid breaking all the code out there, but because we need a consistent 
strategy when resource restrictions that I do think would be viable go in: 
network IO, GPU. It looks like you have left the old constructor -but 
throughout the hadoop codebase moved to an extended one. Not doing that move 
would keep the patch much smaller and reassure me that things are less likely 
to break.


 Add YARN compute units alongside virtual cores
 --

 Key: YARN-1089
 URL: https://issues.apache.org/jira/browse/YARN-1089
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: YARN-1089-1.patch, YARN-1089.patch


 Based on discussion in YARN-1024, we will add YARN compute units as a 
 resource for requesting and scheduling CPU processing power.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1232) Configuration support for RM HA


[ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777932#comment-13777932
 ] 

Karthik Kambatla commented on YARN-1232:


The patch adopts an approach very similar to HDFS. When HA is enabled, the 
following confs are expected:
# {{yarn.resourcemanager.ha.enabled}} set to true
# {{yarn.resourcemanager.address}} set to LOGICAL_NAME
# {{yarn.resourcemanager.ha.nodes.LOGICAL_NAME}} should list the RM ensemble. 
(e.g. rm1,rm2)

Individual RM-specific configs that provide the RPC addresses are:
# One can set {{yarn.resourcemanager.ha.nodes.id}} on individual hosts to 
specify which node in the ensemble, this node corresponds to. (e.g. rm1)
# {{yarn.resourcemanager.rpc-suffix.LOGICAL_NAME.node-id}} where 
{{rpc-address-conf}} comes from {address, scheduler.address, admin.address, 
resource-tracker.address, webapp.address} and {{node-id}} comes from the RM 
ensemble ({rm1, rm2})

Implementation details: All the individual RPC services continue to use the 
configs they use today. When HA is enabled, RMHAProtocolService sets these 
configs from the newly added configs.

Verified the patch manually (pseudo-dist mode) with both today's configs and 
newly added configs.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-899) Get queue administration ACLs working

2013-09-25 Thread Xuan Gong (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778001#comment-13778001
 ] 

Xuan Gong commented on YARN-899:


bq.Is the following code necessary?

Will remove

bq.Why not defining checkAccess in YarnScheduler? I saw other ACL related 
method is defined there as well.

Will move to YarnScheduler

bq.checkViewAccess is used in getApplications and getApplicationReport, does 
|| mean the user doesn't need to have the access to the application whenever 
it has the access to the queue?

For the view access, we can check the union of QueueACLs and ApplicationACLs

 Get queue administration ACLs working
 -

 Key: YARN-899
 URL: https://issues.apache.org/jira/browse/YARN-899
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
Assignee: Xuan Gong
 Attachments: YARN-899.1.patch, YARN-899.2.patch, YARN-899.3.patch, 
 YARN-899.4.patch, YARN-899.5.patch, YARN-899.5.patch, YARN-899.6.patch, 
 YARN-899.7.patch, YARN-899.8.patch


 The Capacity Scheduler documents the 
 yarn.scheduler.capacity.root.queue-path.acl_administer_queue config option 
 for controlling who can administer a queue, but it is not hooked up to 
 anything.  The Fair Scheduler could make use of a similar option as well.  
 This is a feature-parity regression from MR1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1239) Save version information in the state store

Bikas Saha created YARN-1239:


 Summary: Save version information in the state store
 Key: YARN-1239
 URL: https://issues.apache.org/jira/browse/YARN-1239
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Jian He


When creating root dir for the first time we should write version 1. If root 
dir exists then we should check that the version in the state store matches the 
version from config.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception


 [ 
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Omkar Vinit Joshi updated YARN-1219:


Affects Version/s: 2.1.2-beta

 FSDownload changes file suffix making FileUtil.unTar() throw exception
 --

 Key: YARN-1219
 URL: https://issues.apache.org/jira/browse/YARN-1219
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
 Attachments: YARN-1219.patch


 While running a Hive join operation on Yarn, I saw exception as described 
 below. This is caused by FSDownload copy the files into a temp file and 
 change the suffix into .tmp before unpacking it. In unpack(), it uses 
 FileUtil.unTar() which will determine if the file is gzipped by looking at 
 the file suffix:
 {code}
 boolean gzipped = inFile.toString().endsWith(gz);
 {code}
 To fix this problem, we can remove the .tmp in the temp file name.
 Here is the detailed exception:
 org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
   at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
   at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
   at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1215) Yarn URL should include userinfo


[ 
https://issues.apache.org/jira/browse/YARN-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778076#comment-13778076
 ] 

Bikas Saha commented on YARN-1215:
--

Do we need a PB for URL. Can we not serialize URL as a string?

 Yarn URL should include userinfo
 

 Key: YARN-1215
 URL: https://issues.apache.org/jira/browse/YARN-1215
 Project: Hadoop YARN
  Issue Type: Bug
  Components: api
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
 Attachments: YARN-1215-trunk.2.patch, YARN-1215-trunk.patch


 In the {{org.apache.hadoop.yarn.api.records.URL}} class, we don't have an 
 userinfo as part of the URL. When converting a {{java.net.URI}} object into 
 the YARN URL object in {{ConverterUtils.getYarnUrlFromURI()}} method, we will 
 set uri host as the url host. If the uri has a userinfo part, the userinfo is 
 discarded. This will lead to information loss if the original uri has the 
 userinfo, e.g. foo://username:passw...@example.com will be converted to 
 foo://example.com and username/password information is lost during the 
 conversion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1240) mapreduce.jobhistory.webapp.https.address property is not respected

2013-09-25 Thread Yesha Vora (JIRA)

Yesha Vora created YARN-1240:


 Summary: mapreduce.jobhistory.webapp.https.address property is not 
respected
 Key: YARN-1240
 URL: https://issues.apache.org/jira/browse/YARN-1240
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora


The jobhistory server starts on port defined by 
mapreduce.jobhistory.webapp.address property instead 
mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (YARN-1240) mapreduce.jobhistory.webapp.https.address property is not respected


 [ 
https://issues.apache.org/jira/browse/YARN-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Omkar Vinit Joshi reassigned YARN-1240:
---

Assignee: Omkar Vinit Joshi

 mapreduce.jobhistory.webapp.https.address property is not respected
 ---

 Key: YARN-1240
 URL: https://issues.apache.org/jira/browse/YARN-1240
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi

 The jobhistory server starts on port defined by 
 mapreduce.jobhistory.webapp.address property instead 
 mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1240) mapreduce.jobhistory.webapp.https.address property is not respected


 [ 
https://issues.apache.org/jira/browse/YARN-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Omkar Vinit Joshi updated YARN-1240:


Priority: Blocker  (was: Major)

 mapreduce.jobhistory.webapp.https.address property is not respected
 ---

 Key: YARN-1240
 URL: https://issues.apache.org/jira/browse/YARN-1240
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
Priority: Blocker

 The jobhistory server starts on port defined by 
 mapreduce.jobhistory.webapp.address property instead 
 mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1240) mapreduce.jobhistory.webapp.https.address property is not respected


[ 
https://issues.apache.org/jira/browse/YARN-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778156#comment-13778156
 ] 

Omkar Vinit Joshi commented on YARN-1240:
-

looks like few things were missed in YARN-1204. attaching the patch which fixes 
this.

 mapreduce.jobhistory.webapp.https.address property is not respected
 ---

 Key: YARN-1240
 URL: https://issues.apache.org/jira/browse/YARN-1240
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi

 The jobhistory server starts on port defined by 
 mapreduce.jobhistory.webapp.address property instead 
 mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster


[ 
https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778162#comment-13778162
 ] 

Vinod Kumar Vavilapalli commented on YARN-49:
-

TestDistributedShell is sometimes running into YARN-1070, a race condition. It 
passes sometimes and fails sometimes on my box; not related to this patch 
directly.

 Improve distributed shell application to work on a secure cluster
 -

 Key: YARN-49
 URL: https://issues.apache.org/jira/browse/YARN-49
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: applications/distributed-shell
Reporter: Hitesh Shah
Assignee: Vinod Kumar Vavilapalli
 Attachments: YARN-49-20130923.3.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster


[ 
https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778163#comment-13778163
 ] 

Vinod Kumar Vavilapalli commented on YARN-49:
-

Oh, and I just tested this in a multiple nodes' secure setup, all is well!

 Improve distributed shell application to work on a secure cluster
 -

 Key: YARN-49
 URL: https://issues.apache.org/jira/browse/YARN-49
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: applications/distributed-shell
Reporter: Hitesh Shah
Assignee: Vinod Kumar Vavilapalli
 Attachments: YARN-49-20130923.3.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1221) With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely

2013-09-25 Thread Siqi Li (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siqi Li updated YARN-1221:
--

Attachment: YARN1221_v1.patch.txt

 With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely
 -

 Key: YARN-1221
 URL: https://issues.apache.org/jira/browse/YARN-1221
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
 Attachments: YARN1221_v1.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1215) Yarn URL should include userinfo

2013-09-25 Thread Chuan Liu (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778164#comment-13778164
]

Chuan Liu commented on YARN-1215:
-

Thanks for the suggestion!

Do we need a PB for URL. Can we not serialize URL as a string?
URLProto already exists and is used in LocalResourceProto message in
yarn_protos.proto
I don't want to break the wire compatibility because .proto files are already
marked stable in HADOOP-9619.
The current approach should cover most uses cases from my read of Java URL API
below.
http://docs.oracle.com/javase/7/docs/api/java/net/URL.html

Yarn URL should include userinfo

Key: YARN-1215
URL: https://issues.apache.org/jira/browse/YARN-1215
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 3.0.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: YARN-1215-trunk.2.patch, YARN-1215-trunk.patch

In the {{org.apache.hadoop.yarn.api.records.URL}} class, we don't have an
userinfo as part of the URL. When converting a {{java.net.URI}} object into
the YARN URL object in {{ConverterUtils.getYarnUrlFromURI()}} method, we will
set uri host as the url host. If the uri has a userinfo part, the userinfo is
discarded. This will lead to information loss if the original uri has the
userinfo, e.g. foo://username:passw...@example.com will be converted to
foo://example.com and username/password information is lost during the
conversion.

[jira] [Commented] (YARN-1221) With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely


[ 
https://issues.apache.org/jira/browse/YARN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778167#comment-13778167
 ] 

Hadoop QA commented on YARN-1221:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12605119/YARN1221_v1.patch.txt
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2017//console

This message is automatically generated.

 With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely
 -

 Key: YARN-1221
 URL: https://issues.apache.org/jira/browse/YARN-1221
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
 Attachments: YARN1221_v1.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1221) With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely


[ 
https://issues.apache.org/jira/browse/YARN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778174#comment-13778174
 ] 

Sandy Ryza commented on YARN-1221:
--

{code}
-this.totalMB = availableMB + reservedMB + allocatedMB;
+this.totalMB = availableMB;
{code}
Total MB should still include allocatedMB.  I agree that reservedMB should be 
removed from it, but I think this is work for a separate JIRA.  This one is for 
dealing with why reservedMB is calculated incorrectly.

{code}
-  getMetrics().reserveResource(app.getUser(),
-  container.getResource());
{code}
Can you explain the rationale behind removing this?

 With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely
 -

 Key: YARN-1221
 URL: https://issues.apache.org/jira/browse/YARN-1221
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
 Attachments: YARN1221_v1.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster

2013-09-25 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778173#comment-13778173
 ] 

Hitesh Shah commented on YARN-49:
-

+1. Looks good. 

 Improve distributed shell application to work on a secure cluster
 -

 Key: YARN-49
 URL: https://issues.apache.org/jira/browse/YARN-49
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: applications/distributed-shell
Reporter: Hitesh Shah
Assignee: Vinod Kumar Vavilapalli
 Attachments: YARN-49-20130923.3.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1221) With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely

2013-09-25 Thread Siqi Li (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778196#comment-13778196
 ] 

Siqi Li commented on YARN-1221:
---

{code}
-  getMetrics().reserveResource(app.getUser(),
-  container.getResource());
{code}

The reason I removed the code above is that there is no corresponding
unreserve method got called. In addition, app.getUser() and
application.getUser() return the same user.

Maybe we could add getMetrics().unreserveResource(app.getUser(),
container.getResource());
into corresponding unreserve block.

As far as I saw from the webUI, the available memory never get decremented
when it allocates memory to mr jobs. It actually reflects the totally
memory






 With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely
 -

 Key: YARN-1221
 URL: https://issues.apache.org/jira/browse/YARN-1221
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
 Attachments: YARN1221_v1.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (YARN-1242) AHS's resource needs be added to RM's classpath

2013-09-25 Thread Zhijie Shen (JIRA)

Zhijie Shen created YARN-1242:
-

 Summary: AHS's resource needs be added to RM's classpath
 Key: YARN-1242
 URL: https://issues.apache.org/jira/browse/YARN-1242
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal


Maybe we should include AHS classes as well (for developer usage) in yarn and 
yarn.cmd

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1221) With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely


[ 
https://issues.apache.org/jira/browse/YARN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778208#comment-13778208
 ] 

Sandy Ryza commented on YARN-1221:
--

bq. The reason I removed the code above is that there is no corresponding 
unreserve method got called.
Good catch.  But that shouldn't affect the amount shown in the web UI, because 
the metrics for which there is double counting are the leaf queue metrics, 
whereas the value in the web UI is based only off of the root queue metrics.  
Is that not right?

bq. As far as I saw from the webUI, the available memory never get decremented 
when it allocates memory to mr jobs.
Where are you seeing the available memory reported on the web UI?

To be clear, I'm referring to what's shown under the Cluster Metrics section 
when you go to http://rmhost:port/cluster

 With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely
 -

 Key: YARN-1221
 URL: https://issues.apache.org/jira/browse/YARN-1221
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager, scheduler
Affects Versions: 2.1.0-beta
Reporter: Sandy Ryza
 Attachments: YARN1221_v1.patch.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1240) mapreduce.jobhistory.webapp.https.address property is not respected


 [ 
https://issues.apache.org/jira/browse/YARN-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Omkar Vinit Joshi updated YARN-1240:


Attachment: YARN-1240.20131025.1.patch

 mapreduce.jobhistory.webapp.https.address property is not respected
 ---

 Key: YARN-1240
 URL: https://issues.apache.org/jira/browse/YARN-1240
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.1.1-beta
Reporter: Yesha Vora
Assignee: Omkar Vinit Joshi
Priority: Blocker
 Attachments: YARN-1240.20131025.1.patch


 The jobhistory server starts on port defined by 
 mapreduce.jobhistory.webapp.address property instead 
 mapreduce.jobhistory.webapp.https.address when hadoop.ssl.enabled=true.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-49) Improve distributed shell application to work on a secure cluster


[ 
https://issues.apache.org/jira/browse/YARN-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778242#comment-13778242
 ] 

Hudson commented on YARN-49:


SUCCESS: Integrated in Hadoop-trunk-Commit #4468 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/4468/])
YARN-49. Improve distributed shell application to work on a secure cluster. 
Contributed by Vinod Kumar Vavilapalli. (hitesh: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526330)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java


 Improve distributed shell application to work on a secure cluster
 -

 Key: YARN-49
 URL: https://issues.apache.org/jira/browse/YARN-49
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: applications/distributed-shell
Reporter: Hitesh Shah
Assignee: Vinod Kumar Vavilapalli
 Fix For: 2.1.2-beta

 Attachments: YARN-49-20130923.3.txt




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception

2013-09-25 Thread shanyu zhao (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shanyu zhao updated YARN-1219:
--

Fix Version/s: 2.1.2-beta

 FSDownload changes file suffix making FileUtil.unTar() throw exception
 --

 Key: YARN-1219
 URL: https://issues.apache.org/jira/browse/YARN-1219
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
 Fix For: 2.1.2-beta

 Attachments: YARN-1219.patch


 While running a Hive join operation on Yarn, I saw exception as described 
 below. This is caused by FSDownload copy the files into a temp file and 
 change the suffix into .tmp before unpacking it. In unpack(), it uses 
 FileUtil.unTar() which will determine if the file is gzipped by looking at 
 the file suffix:
 {code}
 boolean gzipped = inFile.toString().endsWith(gz);
 {code}
 To fix this problem, we can remove the .tmp in the temp file name.
 Here is the detailed exception:
 org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
   at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
   at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
   at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
   at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:722)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application

2013-09-25 Thread Tassapol Athiapinya (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tassapol Athiapinya updated YARN-1157:
--

Priority: Critical  (was: Major)

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Priority: Critical
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application


 [ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-1157:
--

Attachment: YARN-1157.7.patch

Same patch with javadoc fixes and a method rename.

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
Priority: Critical
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application

2013-09-25 Thread Tassapol Athiapinya (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tassapol Athiapinya updated YARN-1157:
--

Priority: Major  (was: Critical)

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1232) Configuration support for RM HA


 [ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-1232:
---

Attachment: yarn-1232-4.patch

Minor changes - method renaming and typo fixes.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch, 
 yarn-1232-4.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1028) Add FailoverProxyProvider like capability to RMProxy


 [ 
https://issues.apache.org/jira/browse/YARN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated YARN-1028:
---

Attachment: yarn-1028-draft-cumulative.patch

Uploading an initial draft of the patch that includes YARN-1232. Outline:
# ConfiguredFailoverProxyProvider toggles the RM node to connect to.
# ClientRMProxy and ServerRMProxy use this to provide proxy for the respective 
protocols.

 Add FailoverProxyProvider like capability to RMProxy
 

 Key: YARN-1028
 URL: https://issues.apache.org/jira/browse/YARN-1028
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Karthik Kambatla
 Attachments: yarn-1028-draft-cumulative.patch


 RMProxy layer currently abstracts RM discovery and implements it by looking 
 up service information from configuration. Motivated by HDFS and using 
 existing classes from Common, we can add failover proxy providers that may 
 provide RM discovery in extensible ways.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception

2013-09-25 Thread shanyu zhao (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778330#comment-13778330
]

shanyu zhao commented on YARN-1219:
---

The impact of this bug is that Yarn will fail to download any .tar.gz or .tgz
files as resources on Windows platform.

[~ojoshi] Thank you for reviewing this patch. I think the .tmp file name
suffix is only used to make sure the temporary archive file use a different
name than the archive file name itself. So the file name of dCopy has to be
different from sCopy. Because the final unarchived files need to be put in a
folder whose name is the exactly the original archive file name.

Here is what happens when a container downloads an archived resource file,
let's say the file name is stage-3.tar.gz and the destination dir is
filecache/10.
a. Create destDirPath folder filecache/10
b. Create dst_work folder filecache/10_tmp
c. Copy archive file to dTmp filecache/10_tmp/stage-3.tar.gz.tmp
d. Unpack dTmp filecache/10_tmp/stage-3.tar.gz.tmp into dFinal
filecache/10_tmp/stage-3.tar.gz/
e. Rename folder dst_work filecache/10_tmp to destDirPath filecache/10
So after all these steps, the unzipped unarchived files can be found in
filecache/10/stage-3.tar.gz/*

I also checked in Hadoop 1.x the file name being downloaded from HDFS does not
have the .tmp suffix. It is using the original file name as the file name to
be downloaded.

I didn't see anywhere in code to treat the .tmp file differently. If you know
please let me know. If the original author only used a suffix to make sure the
name is different than the original file name, it doesn't seem to be worth it
to add an unnecessary and error-prone rename operations just to keep the
temporary file name suffix.

FSDownload changes file suffix making FileUtil.unTar() throw exception
--

Key: YARN-1219
URL: https://issues.apache.org/jira/browse/YARN-1219
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
Reporter: shanyu zhao
Assignee: shanyu zhao
Fix For: 2.1.2-beta

Attachments: YARN-1219.patch

While running a Hive join operation on Yarn, I saw exception as described
below. This is caused by FSDownload copy the files into a temp file and
change the suffix into .tmp before unpacking it. In unpack(), it uses
FileUtil.unTar() which will determine if the file is gzipped by looking at
the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith(gz);
{code}
To fix this problem, we can remove the .tmp in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

[jira] [Updated] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application

2013-09-25 Thread Xuan Gong (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-1157:


Attachment: YARN-1157.8.patch

fix -1 on javadoc warning

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch, YARN-1157.8.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application


 [ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-1157:
--

Attachment: YARN-1157.8.patch

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch, YARN-1157.8.patch, YARN-1157.8.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1232) Configuration support for RM HA


[ 
https://issues.apache.org/jira/browse/YARN-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778356#comment-13778356
 ] 

Hadoop QA commented on YARN-1232:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12605148/yarn-1232-4.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2020//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2020//console

This message is automatically generated.

 Configuration support for RM HA
 ---

 Key: YARN-1232
 URL: https://issues.apache.org/jira/browse/YARN-1232
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: resourcemanager
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
  Labels: ha
 Attachments: yarn-1232-1.patch, yarn-1232-2.patch, yarn-1232-3.patch, 
 yarn-1232-4.patch


 We should augment the configuration to allow users specify two RMs and the 
 individual RPC addresses for them. This blocks 
 ConfiguredFailoverProxyProvider.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application


[ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778363#comment-13778363
 ] 

Hadoop QA commented on YARN-1157:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12605160/YARN-1157.8.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-YARN-Build/2021//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2021//console

This message is automatically generated.

 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch, YARN-1157.8.patch, YARN-1157.8.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1032) NPE in RackResolve


 [ 
https://issues.apache.org/jira/browse/YARN-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-1032:
--

Assignee: Lohit Vijayarenu

 NPE in RackResolve
 --

 Key: YARN-1032
 URL: https://issues.apache.org/jira/browse/YARN-1032
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.0.5-alpha
 Environment: linux
Reporter: Lohit Vijayarenu
Assignee: Lohit Vijayarenu
Priority: Critical
 Attachments: YARN-1032.1.patch, YARN-1032.2.patch


 We found a case where our rack resolve script was not returning rack due to 
 problem with resolving host address. This exception was see in 
 RackResolver.java as NPE, ultimately caught in RMContainerAllocator. 
 {noformat}
 2013-08-01 07:11:37,708 ERROR [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN 
 CONTACTING RM. 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:99)
   at 
 org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:92)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignMapsWithLocality(RMContainerAllocator.java:1039)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignContainers(RMContainerAllocator.java:925)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:861)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$400(RMContainerAllocator.java:681)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:219)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:243)
   at java.lang.Thread.run(Thread.java:722)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (YARN-1032) NPE in RackResolve


 [ 
https://issues.apache.org/jira/browse/YARN-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-1032:
--

Attachment: YARN-1032.3.patch

Same patch with logging fix.

 NPE in RackResolve
 --

 Key: YARN-1032
 URL: https://issues.apache.org/jira/browse/YARN-1032
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.0.5-alpha
 Environment: linux
Reporter: Lohit Vijayarenu
Assignee: Lohit Vijayarenu
Priority: Critical
 Attachments: YARN-1032.1.patch, YARN-1032.2.patch, YARN-1032.3.patch


 We found a case where our rack resolve script was not returning rack due to 
 problem with resolving host address. This exception was see in 
 RackResolver.java as NPE, ultimately caught in RMContainerAllocator. 
 {noformat}
 2013-08-01 07:11:37,708 ERROR [RMCommunicator Allocator] 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN 
 CONTACTING RM. 
 java.lang.NullPointerException
   at 
 org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:99)
   at 
 org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:92)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignMapsWithLocality(RMContainerAllocator.java:1039)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignContainers(RMContainerAllocator.java:925)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:861)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$400(RMContainerAllocator.java:681)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:219)
   at 
 org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:243)
   at java.lang.Thread.run(Thread.java:722)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1157) ResourceManager UI has invalid tracking URL link for distributed shell application


[ 
https://issues.apache.org/jira/browse/YARN-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13778462#comment-13778462
 ] 

Hudson commented on YARN-1157:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #4470 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/4470/])
YARN-1157. Fixed ResourceManager UI to behave correctly when apps like 
distributed-shell do not set tracking urls. Contributed by Xuan Gong. (vinodkv: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1526371)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RegisterApplicationMasterRequest.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java


 ResourceManager UI has invalid tracking URL link for distributed shell 
 application
 --

 Key: YARN-1157
 URL: https://issues.apache.org/jira/browse/YARN-1157
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Tassapol Athiapinya
Assignee: Xuan Gong
 Fix For: 2.1.2-beta

 Attachments: YARN-1157.1.patch, YARN-1157.2.patch, YARN-1157.2.patch, 
 YARN-1157.3.patch, YARN-1157.4.patch, YARN-1157.5.patch, YARN-1157.6.patch, 
 YARN-1157.7.patch, YARN-1157.8.patch, YARN-1157.8.patch


 Submit YARN distributed shell application. Goto ResourceManager Web UI. The 
 application definitely appears. In Tracking UI column, there will be history 
 link. Click on that link. Instead of showing application master web UI, HTTP 
 error 500 would appear.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (YARN-1032) NPE in RackResolve