date:20120330


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3879:
-

Resolution: Duplicate
Status: Resolved  (was: Patch Available)

Dup of MAPREDUCE-3578.

 yarn script has vestiges of jsvc that need to be cleaned up
 ---

 Key: MAPREDUCE-3879
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3879
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
 Attachments: MAPREDUCE-3879.patch.txt


 If nodemanager is started under root (I know, I know). The following is 
 displayed:
 {noformat}
 /usr/lib/jvm/java-6-sun/bin/java -Dproc_nodemanager -Xmx1000m -jvm server 
 -Dhadoop.log.dir=/var/log/yarn -Dyarn.log.dir=/var/log/yarn 
 -Dhadoop.log.file=yarn-yarn-nodemanager-ahmed-laptop.log 
 -Dyarn.log.file=yarn-yarn-nodemanager-ahmed-laptop.log 
 -Dyarn.home.dir=/usr/lib/hadoop -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop/lib/native 
 -classpath 
 /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/share/hadoop/common/*::/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/:/usr/lib/hadoop/share/hadoop/hdfs/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/lib/*:/etc/hadoop/conf/nm-config/log4j.properties
  org.apache.hadoop.yarn.server.nodemanager.NodeManager
 Unrecognized option: -jvm
 Could not create the Java virtual machine.
 {noformat}
 The culprit is this bit of code that looks suspiciously like what used to be 
 in bin/hdfs in support of jsvc launch:
 {noformat}
 elif [ $COMMAND = nodemanager ] ; then
   CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
   CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
   if [[ $EUID -eq 0 ]]; then
 YARN_OPTS=$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS
   else
 YARN_OPTS=$YARN_OPTS -server $YARN_NODEMANAGER_OPTS
   fi
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

2012-03-30 Thread Ravi Gummadi (Created) (JIRA)

[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi


In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is done 
every time based on the bytes to be written to a distributed cache file. When 
we try to write data to next distributed cache file in the same map task, the 
size of random data generated in each iteration can become small based on the 
particular case. This can make this dist cache data generation slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-4087:


Attachment: 4087.patch

Attaching patch with the fix.

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-4087:


Release Note: Fixes the issue of Generate Dist Cache Data generation job 
slowness.
  Status: Patch Available  (was: Open)

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

2012-03-30 Thread Ravi Gummadi (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242246#comment-13242246
 ] 

Ravi Gummadi commented on MAPREDUCE-4087:
-

Attached patch is for branch-1.

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

2012-03-30 Thread Amar Kamat (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242252#comment-13242252
 ] 

Amar Kamat commented on MAPREDUCE-4087:
---

Looks good to me. +1

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-4087:


Status: Open  (was: Patch Available)

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch, 4087.trunk.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-4087:


Attachment: 4087.trunk.patch

Attaching patch for trunk.

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch, 4087.trunk.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-4087:


Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

 [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
 

 Key: MAPREDUCE-4087
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 4087.patch, 4087.trunk.patch


 In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is 
 done every time based on the bytes to be written to a distributed cache file. 
 When we try to write data to next distributed cache file in the same map 
 task, the size of random data generated in each iteration can become small 
 based on the particular case. This can make this dist cache data generation 
 slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

[
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242275#comment-13242275
]

Hadoop QA commented on MAPREDUCE-4087:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520592/4087.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2115//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2115//console

This message is automatically generated.

[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

Key: MAPREDUCE-4087
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Attachments: 4087.patch, 4087.trunk.patch

In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is
done every time based on the bytes to be written to a distributed cache file.
When we try to write data to next distributed cache file in the same map
task, the size of random data generated in each iteration can become small
based on the particular case. This can make this dist cache data generation
slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4033) MiniMRClientClusterFactory is not setting the temp dir correctly in the conf used to init MiniMRYarnCluster

2012-03-30 Thread Alejandro Abdelnur (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved MAPREDUCE-4033.
---

Resolution: Invalid
  Assignee: Alejandro Abdelnur

Oozie testcases are not using the MiniMR configuration to submit a job, thus 
missing settings shared between MiniMR and the AMs.

Still it seems to be an issue when the staging HDFS directory of the AM and the 
JH differ and the app never ends. will verify and open JIRA if appropriate.

 MiniMRClientClusterFactory is not setting the temp dir correctly in the conf 
 used to init MiniMRYarnCluster
 ---

 Key: MAPREDUCE-4033
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4033
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: minicluster-oozie-pig.txt


 Oozie testcases are failing randomly because MR2 reports the job as unknown.
 This seems to happen when Oozie queries via JobClient.getJob(JOBID) for a 
 JOBID that just finished.
 {code}
 org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job 
 [job_1332176678205_0011] associated with action 
 [000-120319101023910-oozie-tucu-W@pig-action].  Failing this action!
 {code}
 Oozie reports this error when JobClient.getJob(JOBID) returns NULL.
 Looking at the mini cluster logs the job definitely run.
 {code}
  find . -name *1332176678205_0011*
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_02
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_01
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_02
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_01
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_02
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_01
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_02
 ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_01
 {code}
 It seems there is a gap until the the job is avail in the JH server.
 If this gap is unavoidable we need to ensure Oozie always waits at least the 
 gap time before querying for a job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

[
https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242285#comment-13242285
]

Hadoop QA commented on MAPREDUCE-4087:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520595/4087.trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2116//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2116//console

This message is automatically generated.

[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases

[jira] [Resolved] (MAPREDUCE-4037) Fails to start proxy server due to webapps/proxy not found in CLASSPATH

2012-03-30 Thread Devaraj K (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K resolved MAPREDUCE-4037.
--

Resolution: Duplicate
  Assignee: (was: Devaraj K)

Duplicate of MAPREDUCE-3916.

 Fails to start proxy server due to webapps/proxy not found in CLASSPATH
 ---

 Key: MAPREDUCE-4037
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4037
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Devaraj K
Priority: Critical

 {code:xml}
 2012-03-20 10:41:35,688 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
   at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
   at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
   at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
   at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:90)
   at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
   at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 2012-03-20 10:41:35,689 ERROR 
 org.apache.hadoop.yarn.service.CompositeService: Error starting services 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 org.apache.hadoop.yarn.YarnException: Could not start proxy web server
   at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:100)
   at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
   at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 Caused by: java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
   at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
   at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
   at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
   at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:90)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-3916) various issues with running yarn proxyserver

2012-03-30 Thread Devaraj K (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K reassigned MAPREDUCE-3916:


Assignee: Devaraj K

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
Assignee: Devaraj K
 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3916:
-

Attachment: MAPREDUCE-3916.patch

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
Assignee: Devaraj K
 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3916:
-

   Labels: mrv2  (was: )
 Target Version/s: 2.0.0, 3.0.0
Affects Version/s: 3.0.0
   2.0.0
   Status: Patch Available  (was: Open)

{quote}Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
at all correctly disable it and produces a message: 
org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the 
proxy will not run). This contradicts the documentation provided for 
yarn.web-proxy.address in yarn-default.xml{quote}
 If we don't set 'yarn.web-proxy.address', then the proxy server will not start 
as separate process and start up fails with the message 
'org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the 
proxy will not run' and proxy will run as part of the RM.

 If we give the value for 'yarn.web-proxy.address' which matches to the 
'yarn.resourcemanager.address' then proxy server start up fails with Address 
already in use. And also it is not starting as part of the RM. It is not as 
per the documentation. I have modified the description for this property 
accordingly.

{quote}
Setting yarn.web-proxy.address and running the service results in the following:
{quote}

I have added the 'hadoop-yarn-common\src\main\resources\webapps\proxy' dir with 
.keep file in it to resolve this problem as done for other components.

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
  Labels: mrv2
 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at

[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3916:
-

Priority: Critical  (was: Major)
Target Version/s: 2.0.0, 3.0.0  (was: 3.0.0, 2.0.0)

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242313#comment-13242313
 ] 

Hudson commented on MAPREDUCE-3989:
---

Integrated in Hadoop-Hdfs-trunk #1000 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/])
MAPREDUCE-3989. Cap space usage of default log4j rolling policy. 
Contributed by Patrick Hunt (Revision 1307102)

 Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307102
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh


 cap space usage of default log4j rolling policy (mr specific changes)
 -

 Key: MAPREDUCE-3989
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3989.patch


 see HADOOP-8149 for background on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4081) TestMROutputFormat.java does not compile


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242319#comment-13242319
 ] 

Hudson commented on MAPREDUCE-4081:
---

Integrated in Hadoop-Hdfs-trunk #1000 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/])
MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via 
bobby) (Revision 1306901)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1306901
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java


 TestMROutputFormat.java does not compile
 

 Key: MAPREDUCE-4081
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4081
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, mrv2
Affects Versions: 2.0.0, trunk
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4081.patch


 [ERROR] 
 /hadoop/src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java:[36,7]
  class TestConfInCheckSpec is public, should be declared in a file named 
 TestConfInCheckSpec.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver

2012-03-30 Thread Alejandro Abdelnur (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242328#comment-13242328
 ] 

Alejandro Abdelnur commented on MAPREDUCE-3916:
---

+1. built/deployed pseudo cluster, set yarn.web-proxy.address and proxy started 
successfully

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver

[
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242329#comment-13242329
]

Hadoop QA commented on MAPREDUCE-3916:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520600/MAPREDUCE-3916.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//console

This message is automatically generated.

various issues with running yarn proxyserver

Key: MAPREDUCE-3916
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
Labels: mrv2
Attachments: MAPREDUCE-3916.patch

Seem like yarn proxyserver is not operational when running out of the 0.23.1
RC2 tarball.
# Setting yarn.web-proxy.address to match yarn.resourcemanager.address
doesn't disable the proxyserver (althought not setting yarn.web-proxy.address
at all correctly disable it and produces a message:
org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so
the proxy will not run). This contradicts the documentation provided for
yarn.web-proxy.address in yarn-default.xml
# Setting yarn.web-proxy.address and running the service results in the
following:
{noformat}
$ ./sbin/yarn-daemon.sh start proxyserver
starting proxyserver, logging to
/tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
/usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m
-Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs
-Dyarn.log.dir=/tmp/hadoop-0.23.1/logs
-Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log
-Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir=
-Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/tmp/hadoop-0.23.1/lib/native
-Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs
-Dyarn.log.dir=/tmp/hadoop-0.23.1/logs
-Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log
-Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log
-Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA
-Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath
/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
{noformat}
with the following message found in the logs:
{noformat}
2012-02-24 09:26:31,099 FATAL
org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web
server
java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
at
org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
{noformat}

--
This message is automatically

[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver

2012-03-30 Thread Alejandro Abdelnur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-3916:
--

  Resolution: Fixed
   Fix Version/s: 2.0.0
Target Version/s: 2.0.0, 3.0.0  (was: 3.0.0, 2.0.0)
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks Devaraj. I had to manually create the proxy/.keep file as the patch 
command ignored it. Committed to trunk and branch-2

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver

2012-03-30 Thread Devaraj K (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242335#comment-13242335
 ] 

Devaraj K commented on MAPREDUCE-3916:
--

Thanks a lot Alejandro for reviewing and verifying the patch.

{quote}
-1 tests included. The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{quote}

No new tests required because changes done in yarn-default.xml and added one 
dir.

 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242337#comment-13242337
 ] 

Hudson commented on MAPREDUCE-3916:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2029 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2029/])
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via 
tucu) (Revision 1307413)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307413
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242338#comment-13242338
 ] 

Hudson commented on MAPREDUCE-3916:
---

Integrated in Hadoop-Common-trunk-Commit #1954 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1954/])
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via 
tucu) (Revision 1307413)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307413
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242342#comment-13242342
 ] 

Hudson commented on MAPREDUCE-3989:
---

Integrated in Hadoop-Mapreduce-trunk #1035 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/])
MAPREDUCE-3989. Cap space usage of default log4j rolling policy. 
Contributed by Patrick Hunt (Revision 1307102)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307102
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh


 cap space usage of default log4j rolling policy (mr specific changes)
 -

 Key: MAPREDUCE-3989
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3989.patch


 see HADOOP-8149 for background on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4081) TestMROutputFormat.java does not compile


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242357#comment-13242357
 ] 

Hudson commented on MAPREDUCE-4081:
---

Integrated in Hadoop-Mapreduce-trunk #1035 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/])
MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via 
bobby) (Revision 1306901)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1306901
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java


 TestMROutputFormat.java does not compile
 

 Key: MAPREDUCE-4081
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4081
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, mrv2
Affects Versions: 2.0.0, trunk
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Blocker
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4081.patch


 [ERROR] 
 /hadoop/src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java:[36,7]
  class TestConfInCheckSpec is public, should be declared in a file named 
 TestConfInCheckSpec.java

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with 0.20.203.


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242354#comment-13242354
 ] 

Hudson commented on MAPREDUCE-3377:
---

Integrated in Hadoop-Mapreduce-trunk #1035 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/])
MAPREDUCE-3377. Added a unit test to ensure 
OutputCommitter.checkOutputSpecs is called prior to copying job.xml. 
Contributed by Jane Chen. (Revision 1306736)

 Result = SUCCESS
acmurthy : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1306736
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java


 Compatibility issue with 0.20.203.
 --

 Key: MAPREDUCE-3377
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.203.0
 Environment: 0.20.203.0 standalone JDK 1.6 Linux
Reporter: Jane Chen
Assignee: Jane Chen
 Fix For: 1.0.3, 2.0.0

 Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, 
 mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, 
 mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before


 I have an OutputFormat which implements Configurable.  I set new config 
 entries to a job configuration during checkOutputSpec() so that the tasks 
 will get the config entries through the job configuration.  This works fine 
 in 0.20.2, but stopped working starting from 0.20.203.  With 0.20.203, my 
 OutputFormat still has the configuration set, but the copy a task gets does 
 not have the new entries that are set as part of checkOutputSpec().  
 I believe that the problem is with JobClient.  The job configuration needs to 
 wait till checkOutputSpec() is returned before being cloned and submitted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver

2012-03-30 Thread Robert Joseph Evans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242375#comment-13242375
 ] 

Hudson commented on MAPREDUCE-3916:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1967 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1967/])
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via 
tucu) (Revision 1307413)

 Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307413
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml


 various issues with running yarn proxyserver
 

 Key: MAPREDUCE-3916
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager, webapps
Affects Versions: 0.23.1, 2.0.0, 3.0.0
Reporter: Roman Shaposhnik
Assignee: Devaraj K
Priority: Critical
  Labels: mrv2
 Fix For: 2.0.0

 Attachments: MAPREDUCE-3916.patch


 Seem like yarn proxyserver is not operational when running out of the 0.23.1 
 RC2 tarball.
 # Setting yarn.web-proxy.address to match yarn.resourcemanager.address 
 doesn't disable the proxyserver (althought not setting yarn.web-proxy.address 
 at all correctly disable it and produces a message: 
 org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so 
 the proxy will not run). This contradicts the documentation provided for 
 yarn.web-proxy.address in yarn-default.xml
 # Setting yarn.web-proxy.address and running the service results in the 
 following:
 {noformat}
 $ ./sbin/yarn-daemon.sh start proxyserver 
 starting proxyserver, logging to 
 /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out
 /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m 
 -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= 
 -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native 
 -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs 
 -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log 
 -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA 
 -Dyarn.root.logger=INFO,DRFA 
 -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath 
 /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*
  org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer
 {noformat}
 with the following message found in the logs:
 {noformat}
 2012-02-24 09:26:31,099 FATAL 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web 
 server
 java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
 at 
 org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:224)
 at org.apache.hadoop.http.HttpServer.init(HttpServer.java:164)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85)
 at 
 org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
 at 
 org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4085) Kill task attempts longer than a configured queue max time


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242391#comment-13242391
 ] 

Robert Joseph Evans commented on MAPREDUCE-4085:


I can see the need for something like this, to ensure that new jobs can run and 
meet their SLAs, but I think it would be better to have it be part of a 
preemption like mechanism, where we let the tasks run until there is some other 
Task/Container(for MRv2) that is requested.  Once there is a need for those 
resources iff the current task/container has gone over the configured limit the 
JT/RM, on the next heartbeat, can inform the TT/NM to kill the task/container.  
The fair scheduler already supports preemption and perhaps this could be added 
there.

MAPREDUCE-3938 was filed to add preemption to the Capacity Scheduler for 2.0 
and it might be good to add this in as part of the design there.

I don't really like the idea of having a hard limit on the runtime.  What is 
more if there is a hard limit on how long a task can run for I see very little 
benefit in having it rescheduled more then once.  If it was a slow node, then 
OK we can pick another node and it might finish in time, but unless the cluster 
is very heterogeneous the task is just going to run to the maximum time limit 4 
times and then the Job will be failed.  

 Kill task attempts longer than a configured queue max time
 --

 Key: MAPREDUCE-4085
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4085
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: task
Reporter: Allen Wittenauer

 For some environments, it is desirable to have certain queues have an SLA 
 with regards to task turnover.  (i.e., a slot will be free in X minutes and 
 scheduled to the appropriate job)  Queues should have a 'task time limit' 
 that would cause task attempts over this time to be killed. This leaves open 
 the possibility that if the task was on a bad node, it could still be 
 rescheduled up to max.task.attempt times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4085) Kill task attempts longer than a configured queue max time

2012-03-30 Thread Allen Wittenauer (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242427#comment-13242427
 ] 

Allen Wittenauer commented on MAPREDUCE-4085:
-

It seems very wrong to me to include this functionality in a specific scheduler.

The other thing to keep in mind is that at least in our use case, we want to 
punish jobs in that are essentially in the wrong queue.  Pre-emption works out 
to be the incorrect action in this case; we ultimately do want to the job to be 
failed as negative feedback to the user that they did something wrong, 
especially for primarily ad hoc usage (or, as you said, very heterogeneous).

 Kill task attempts longer than a configured queue max time
 --

 Key: MAPREDUCE-4085
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4085
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: task
Reporter: Allen Wittenauer

 For some environments, it is desirable to have certain queues have an SLA 
 with regards to task turnover.  (i.e., a slot will be free in X minutes and 
 scheduled to the appropriate job)  Queues should have a 'task time limit' 
 that would cause task attempts over this time to be killed. This leaves open 
 the possibility that if the task was on a bad node, it could still be 
 rescheduled up to max.task.attempt times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated MAPREDUCE-4079:
---

Attachment: MR-4079-trunk.txt
MR-4079-full-branch-0.23.txt

The trunk patch depends on the patch in HADOOP-8227. I will put this JIRA in
patch available once it is integrated in.

Unfortunetly for 0.23 the RPC engine code is split among the projects so the
patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time. I
have attached the full patch for branch-0.23 for both of these JIRAs. Once
both JIRAs are approved for trunk and branch-2.0, I can put the full patch into
branch-0.23.

Allow MR AppMaster to limit ephemeral port range.
-

Key: MAPREDUCE-4079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am, mrv2
Affects Versions: 0.23.2, 2.0.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt

Having the MapReduce Application Masters bind to any ephemeral port makes it
very difficult to setup ACLs. mapreduce.job.am-access-disabled from
MAPREDUCE-3251 is not a practical permanent solution for all jobs.
Especially for tools like pig where they are not aware of
mapreduce.job.am-access-disabled and may deal with it properly.
We should add in a config option that would allow someone to restrict the
range of ports that the MR-AM can bind to. It will slow down startup in some
cases because we will have to probe for open ports instead of just asking the
OS to find one for us. But we can make that conditional on this config so
users who do not set this config do not see any performance degradation.

[jira] [Created] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing

2012-03-30 Thread Ravi Prakash (Created) (JIRA)

Task stuck in JobLocalizer prevented other tasks on the same node from 
committing
-

 Key: MAPREDUCE-4088
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 0.20.205.0
Reporter: Ravi Prakash
Priority: Critical


We saw that as a result of HADOOP-6963, one task was stuck in this

Thread 23668: (state = IN_NATIVE)
 - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled 
frame; information may be imprecise)
 - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 
(Compiled frame)
 - java.io.File.exists() @bci=20, line=733 (Compiled frame)
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled 
frame)
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
(Compiled frame)
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
(Compiled frame)

 TONS MORE OF THIS SAME LINE
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
(Compiled frame)
.
.
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
(Compiled frame)
 - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
(Interpreted frame)
ne=451 (Interpreted frame)
 - 
org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration,
 java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) 
@bci=150, line=324 (Interpreted frame)
 - 
org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration)
 @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame)
 - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, 
java.lang.String, org.apache.hadoop.fs.Path, 
org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted 
frame)
 - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted 
frame)
 - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted 
frame)
 - 
java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
 java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
 - 
org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
 @bci=14, line=1082 (Interpreted frame)
 - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, 
line=530 (Interpreted frame)

While all other tasks on the same node were stuck in 
Thread 32141: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - 
org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
 org.apache.hadoop.mapred.Task$TaskReporter, 
org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame)
 - 
org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
 org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted 
frame)
 - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, 
org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted 
frame)
 - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame)
 - 
java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
 java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
 - 
org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
 @bci=14, line=1082 (Interpreted frame)
 - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 
(Interpreted frame)

This should never happen. A stuck task should never prevent other tasks from 
different jobs on the same node from committing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-03-30 Thread Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242600#comment-13242600
 ] 

Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059:
---

Hi Robert,
I just have gone through a portion of the patch and have few comments/doubts on 
the patch. 

1. {code:title=CachedHistoryStorage.java|borderStyle=solid}
if(offset == null || offset  0) offset = 0l;
if(count == null) count = Long.MAX_VALUE;

long at = 0;
long end = offset + count - 1;
LOG.error(Looking for entries starting at +offset+ with a length of 
+count+ so ending at +end);
for (Job job : jobs) {
  LOG.error(Looking at job END: +at+ = +end);
  if(at  end) {
break;
  }
{code}

Suppose say, in the above code offset is set as 100 and count is not 
initialized, so count will be initialized as Long.MAX_VALUE.
If so, *end* may get negative value (because of integer overflow) and hence for 
loop will be exited in the first iteration itself.
But I think, we should return entries starting from 100th entry.

Am I missing anything here?

2. I think it is better to add sanity check like this in 
CachedHistoryStorage#getPartialJobs()
{noformat}
if(offset  jobs.size()) {
return allJobs;
}
{noformat}

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Created) (JIRA)

Hung Tasks never time out. 
---

 Key: MAPREDUCE-4089
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker


The AM will timeout a task through mapreduce.task.timeout only when it does not 
hear from the task within the given timeframe.  On 1.0 a task must be making 
progress, either by reading input from HDFS, writing output to HDFS, writing to 
a log, or calling a special method to inform it that it is still making 
progress.

This is because on 0.23 a status update which happens every 3 seconds is 
counted as progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading

2012-03-30 Thread Anupam Seth (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anupam Seth updated MAPREDUCE-4072:
---

Attachment: MAPREDUCE-4072-branch-23.patch

Uploading patch to remove transmission of java.library.path to container.

 User set java.library.path seems to overwrite default creating problems 
 native lib loading
 --

 Key: MAPREDUCE-4072
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Anupam Seth
Assignee: Anupam Seth
 Attachments: MAPREDUCE-4072-branch-23.patch


 This was found by Peeyush Bishnoi.
 While running a distributed cache example with Hadoop-0.23,
 tasks are failing as follows:
 
 Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: at
 org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
 org.apache.hadoop.util.Shell.run(Shell.java:188) at
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
 org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
 java.util.concurrent.FutureTask.run(FutureTask.java:138) at
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
 is user
 
 Same Pig script and command work successfully on 0.20
 See this in the stderr:
 Exception in thread main java.lang.ExceptionInInitializerError
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at
 org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
 at
 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
 at org.apache.hadoop.security.Groups.(Groups.java:54)
 at
 org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
 at
 org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
 at
 org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
 at
 org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
 Caused by: java.lang.RuntimeException: Bailing out since native library
 couldn't be loaded
 at
 org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
 ... 12 more
 Pig command:
 $ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives 
 -Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib
 -Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp 
 -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading

2012-03-30 Thread Anupam Seth (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anupam Seth updated MAPREDUCE-4072:
---

Status: Patch Available  (was: Open)

 User set java.library.path seems to overwrite default creating problems 
 native lib loading
 --

 Key: MAPREDUCE-4072
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Anupam Seth
Assignee: Anupam Seth
 Attachments: MAPREDUCE-4072-branch-23.patch


 This was found by Peeyush Bishnoi.
 While running a distributed cache example with Hadoop-0.23,
 tasks are failing as follows:
 
 Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: at
 org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
 org.apache.hadoop.util.Shell.run(Shell.java:188) at
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
 org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
 java.util.concurrent.FutureTask.run(FutureTask.java:138) at
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
 is user
 
 Same Pig script and command work successfully on 0.20
 See this in the stderr:
 Exception in thread main java.lang.ExceptionInInitializerError
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at
 org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
 at
 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
 at org.apache.hadoop.security.Groups.(Groups.java:54)
 at
 org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
 at
 org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
 at
 org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
 at
 org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
 Caused by: java.lang.RuntimeException: Bailing out since native library
 couldn't be loaded
 at
 org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
 ... 12 more
 Pig command:
 $ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives 
 -Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib
 -Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp 
 -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242620#comment-13242620
]

Robert Joseph Evans commented on MAPREDUCE-4089:

From looking at the code for Task. When TaskReporter.progress() is called,
all it does is set the progress flag, which causes a progress update to be
sent to the AM instead of a ping to be sent. So I would guess is that we want
to stop counting pings when measuring progress in the AM. So it should
probably be a simple 1 line change.

Hung Tasks never time out.
---

Key: MAPREDUCE-4089
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker

The AM will timeout a task through mapreduce.task.timeout only when it does
not hear from the task within the given timeframe. On 1.0 a task must be
making progress, either by reading input from HDFS, writing output to HDFS,
writing to a log, or calling a special method to inform it that it is still
making progress.
This is because on 0.23 a status update which happens every 3 seconds is
counted as progress.

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-03-30 Thread Robert Joseph Evans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242632#comment-13242632
 ] 

Robert Joseph Evans commented on MAPREDUCE-4059:


That is a great catch.  I did totally miss the possibility of int overflow, and 
with count being negative. Will update the patch.  Thanks.

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4059:
---

Status: Open  (was: Patch Available)

Addressing review comments.

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface

2012-03-30 Thread Robert Joseph Evans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242635#comment-13242635
 ] 

Robert Joseph Evans commented on MAPREDUCE-4059:


Oh I should also remove my debugging log messages that are logged at error!!

 The history server should have a separate pluggable storage/query interface
 ---

 Key: MAPREDUCE-4059
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-4059.txt, MR-4059.txt


 The history server currently caches all parsed jobs in RAM.  These jobs can 
 be very large because of counters.  It would be nice to have a pluggable 
 interface for the cacheing and querying of the cached data so that we can 
 play around with different implementations.  Also just for cleanness of the 
 code it would be nice to split the very large JobHistoryServer.java into a 
 few smaller ones that are more understandable and readable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading

2012-03-30 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242657#comment-13242657
]

Hadoop QA commented on MAPREDUCE-4072:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12520642/MAPREDUCE-4072-branch-23.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//console

This message is automatically generated.

User set java.library.path seems to overwrite default creating problems
native lib loading
--

Key: MAPREDUCE-4072
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Anupam Seth
Assignee: Anupam Seth
Attachments: MAPREDUCE-4072-branch-23.patch

This was found by Peeyush Bishnoi.
While running a distributed cache example with Hadoop-0.23,
tasks are failing as follows:

Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: at
org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
org.apache.hadoop.util.Shell.run(Shell.java:188) at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
java.util.concurrent.FutureTask.run(FutureTask.java:138) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
is user

Same Pig script and command work successfully on 0.20
See this in the stderr:
Exception in thread main java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
at org.apache.hadoop.security.Groups.(Groups.java:54)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
at
org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
Caused by: java.lang.RuntimeException: Bailing out since native library
couldn't be loaded
at
org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
... 12 more
Pig command:
$ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives
-Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib
-Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp
-Dmapred.create.symlink=yes

[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated MAPREDUCE-4089:
---

Attachment: MR-4089.txt

This patch addresses the timeout issue, and it does so by making ping not
update progress. It is still not completely compatible with 1.0, as in 1.0 if
the timeout is set to 0 the task will never timeout. But because this patch
makes it so ping is ignored a task that has a timeout of 0, but is so locked up
that it cannot ping anymore will never timeout.

I am planning to address these in a follow on JIRA, unless someone has some
objections to doing so.

I also have not run all of the unit tests yet.

Hung Tasks never time out.
---

[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4089:
---

Target Version/s: 0.23.2, 2.0.0, trunk  (was: trunk, 2.0.0, 0.23.2)
  Status: Patch Available  (was: Open)

 Hung Tasks never time out. 
 ---

 Key: MAPREDUCE-4089
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
 Attachments: MR-4089.txt


 The AM will timeout a task through mapreduce.task.timeout only when it does 
 not hear from the task within the given timeframe.  On 1.0 a task must be 
 making progress, either by reading input from HDFS, writing output to HDFS, 
 writing to a log, or calling a special method to inform it that it is still 
 making progress.
 This is because on 0.23 a status update which happens every 3 seconds is 
 counted as progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading

2012-03-30 Thread Kihwal Lee (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242671#comment-13242671
 ] 

Kihwal Lee commented on MAPREDUCE-4072:
---

Users should not include -Djava.library.path in task jvm options. The code 
change looks good, but we need to make sure this is documented somewhere as an 
incompatible change. 

 User set java.library.path seems to overwrite default creating problems 
 native lib loading
 --

 Key: MAPREDUCE-4072
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Anupam Seth
Assignee: Anupam Seth
 Attachments: MAPREDUCE-4072-branch-23.patch


 This was found by Peeyush Bishnoi.
 While running a distributed cache example with Hadoop-0.23,
 tasks are failing as follows:
 
 Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: at
 org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at
 org.apache.hadoop.util.Shell.run(Shell.java:188) at
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at
 org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241)
 at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
 java.util.concurrent.FutureTask.run(FutureTask.java:138) at
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user
 is user
 
 Same Pig script and command work successfully on 0.20
 See this in the stderr:
 Exception in thread main java.lang.ExceptionInInitializerError
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at
 org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179)
 at
 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
 at org.apache.hadoop.security.Groups.(Groups.java:54)
 at
 org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178)
 at
 org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
 at
 org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
 at
 org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75)
 Caused by: java.lang.RuntimeException: Bailing out since native library
 couldn't be loaded
 at
 org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48)
 ... 12 more
 Pig command:
 $ pig -Dmapred.job.queue.name=queue -Dmapred.cache.archives=archives 
 -Dmapred.child.java.opts=-Djava.library.path=./ygeo/lib
 -Dip2geo.preLoadLibraries=some other libs -Djava.io.tmpdir=/grid/0/tmp 
 -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242675#comment-13242675
 ] 

Robert Joseph Evans commented on MAPREDUCE-4089:


OK Talking to people here it looks like there are a significant number of users 
that I know of that set the timeout to 0, so I am going to come up with a new 
patch to have the 0 timeout be acceptable too.

 Hung Tasks never time out. 
 ---

 Key: MAPREDUCE-4089
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
 Attachments: MR-4089.txt


 The AM will timeout a task through mapreduce.task.timeout only when it does 
 not hear from the task within the given timeframe.  On 1.0 a task must be 
 making progress, either by reading input from HDFS, writing output to HDFS, 
 writing to a log, or calling a special method to inform it that it is still 
 making progress.
 This is because on 0.23 a status update which happens every 3 seconds is 
 counted as progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters

2012-03-30 Thread Ahmed Radwan (Created) (JIRA)

Branch 1 pipes doesn't work on MR2 clusters
---

 Key: MAPREDUCE-4090
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Ahmed Radwan


If I compile pipes examples on branch 1: 

{code}
ant -Dcompile.c++=yes examples
{code}

And then try to run it on an MR2 cluster, the pipes job hangs forever at map 0% 
reduce 0%, I can see in the maps stderr:

{code}
Hadoop Pipes Exception: Illegal text protocol command 
{code}

The issue here is that if users have older pipes job, they won't be able to run 
it on MR2 (without recompilation). Is this expected or there is something to be 
fixed so jobs can be used interchangeably? Or should we document it as an 
incompatibility?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.

[
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242705#comment-13242705
]

Hadoop QA commented on MAPREDUCE-4089:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520653/MR-4089.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2119//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2119//console

This message is automatically generated.

Hung Tasks never time out.
---

[jira] [Created] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Alejandro Abdelnur (Created) (JIRA)

tools testcases failing because of MAPREDUCE-4082
-

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0


MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
classpath to the test classpath.

The objective of MAPREDUCE-4082 was to remove the generated-classpath file from 
the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
test-classpath would do the trick.

This is breaking tools testcases (most likely) because of different classloader 
being used by maven for main classpath and test classpath.



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Alejandro Abdelnur (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242749#comment-13242749
]

Alejandro Abdelnur commented on MAPREDUCE-4091:
---

The generated-classpath file has a meaning for running testcases in the current
project. If downstream projects want to leverage that file, they have create it
in their project (i.e. oozie-core creates one to be able to use MiniMR).

Having the generated-classpath file bundled in the JAR is pointless, and as
MAPREDUCE-4082 stated, pollutes the classpath big time.

An alternate way to achieve not having the generated-classpath file in the JAR
is to excluded it in the maven-jar-plugin.

tools testcases failing because of MAPREDUCE-4082
-

Key: MAPREDUCE-4091
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
Fix For: 2.0.0

MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main
classpath to the test classpath.
The objective of MAPREDUCE-4082 was to remove the generated-classpath file
from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the
test-classpath would do the trick.
This is breaking tools testcases (most likely) because of different
classloader being used by maven for main classpath and test classpath.

[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Alejandro Abdelnur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4091:
--

Attachment: MAPREDUCE-4091.patch

the tools (streaming  archives) failing testcases are passing again.

and the hadoop-mapreduce-client-app JAR does not have the 
mrapp-generated-classpath file in it.

 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4062) AM Launcher thread can hang forever

2012-03-30 Thread Jason Lowe (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242754#comment-13242754
 ] 

Jason Lowe commented on MAPREDUCE-4062:
---

We've seen an issue where using both an RPC-level timeout (in this case the 
ping timeout) and the timer task can cause the AM to lose track of a container 
and hang the job.  Here's the relevant part of the AM log:

{noformat}
2012-03-29 07:32:17,794 ERROR [ContainerLauncher #199] 
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException 
as:container_1333003059741_0010_01_003408 (auth:SIMPLE) 
cause:java.net.SocketTimeoutException: 6 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/x.x.x.x:x remote=x.x.x.x.x/x.x.x.x:x]
2012-03-29 07:32:17,794 WARN [ContainerLauncher #199] 
org.apache.hadoop.ipc.Client: Exception encountered while connecting to the 
server : java.net.SocketTimeoutException: 6 millis timeout while waiting 
for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/x.x.x.x:x 
remote=x.x.x.x.x/x.x.x.x:x]
2012-03-29 07:32:17,794 ERROR [ContainerLauncher #199] 
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException 
as:container_1333003059741_0010_01_003408 (auth:SIMPLE) 
cause:java.io.IOException: java.net.SocketTimeoutException: 6 millis 
timeout while waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/x.x.x.x:x 
remote=x.x.x.x.x/x.x.x.x:x]
2012-03-29 07:32:17,795 WARN [Timer-1] 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Couldn't 
complete CONTAINER_REMOTE_CLEANUP on 
container_1333003059741_0010_01_003408/attempt_1333003059741_0010_m_003097_0. 
Interrupting and returning
2012-03-29 07:32:17,798 INFO [Timer-1] 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Stack trace 
of the command-thread: 

at java.util.Arrays.copyOf(Arrays.java:2882)
at 
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at 
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at java.lang.StackTraceElement.toString(StackTraceElement.java:157)
at java.lang.String.valueOf(String.java:2826)
at java.lang.StringBuilder.append(StringBuilder.java:115)
at java.lang.Throwable.printStackTrace(Throwable.java:512)
at 
org.apache.hadoop.util.StringUtils.stringifyException(StringUtils.java:64)
at 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:260)
at 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:479)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2012-03-29 07:32:17,800 WARN [ContainerLauncher #199] 
org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted
java.lang.InterruptedException
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
at 
java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
at 
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:179)
at 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:263)
at 
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:479)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}

Looks like the socket timeout and the timer task timeout occurred almost 
simultaneously.  The socket exception was caught first, and during the catch 
clause we fielded the interrupted exception.  That broke us out of the handling 
of the socket exception and we never marked the container status properly 
before leaving.

So I'm +1 on removing the timer task and relying on the rpcTimeout, unless 
there are other cases besides RPC where we'd expect the container launcher to 
get stuck.

 AM Launcher thread can hang forever
 ---

 Key: MAPREDUCE-4062
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4062
 Project: Hadoop Map/Reduce

[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Alejandro Abdelnur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4091:
--

Status: Patch Available  (was: Open)

 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Mahadev konar (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242760#comment-13242760
 ] 

Mahadev konar commented on MAPREDUCE-4091:
--

+1 for the patch. Looks good.

 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

2012-03-30 Thread Alejandro Abdelnur (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4091:
--

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

committed to trunk  branch-2

 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs

2012-03-30 Thread Siddharth Seth (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242773#comment-13242773
]

Siddharth Seth commented on MAPREDUCE-4067:
---

YarnRemoteException was serializing the actual exception and sending it over
the wire. It was missing an unwind method to get the actual exception. IAC -
clients should not be expected to unwind and that should've been handled in the
RPC layer itself.

Instead of having the APIs throw an IOException, I'd prefer to define a Yarn /
HadoopException which extends Exception and can later be sub-classed for more
specific exceptions. Also, possibly add error codes to these exceptions as well
as to what is sent over the wire. That may make things easier for clients
written in other languages.

Replace YarnRemoteException with IOException in MRv2 APIs
-

Key: MAPREDUCE-4067
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Siddharth Seth
Priority: Blocker

YarnRemoteException is defined as a generic wrapper for all the exceptions in
yarn. I think we should instead throw IOExceptions in the API, which can
later be extended for more specialized exceptions without breaking
compatibility.

[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242772#comment-13242772
 ] 

Hudson commented on MAPREDUCE-4082:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2031 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2031/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in 
 the module JAR
 

 Key: MAPREDUCE-4082
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4082.patch


 Currently the mrapp-generated-classpath file containing the 'built' 
 classpath, which only makes sense during building/testing in the machine 
 where the build happens, is bundled in the hadoop-mapreduce-client-app JAR.
 Because the file is bundled in the hadoop-mapreduce-client-app JAR, its 
 contents are added to the classpath of all MR jobs. 
 All this entries are useless and just pollute the classpath.
 This file should not be bundled in the hadoop-mapreduce-client-app JAR.
 As an example, the contents of this file in my local built are:
 {code}

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242771#comment-13242771
 ] 

Hudson commented on MAPREDUCE-4091:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2031 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2031/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242775#comment-13242775
 ] 

Hudson commented on MAPREDUCE-4091:
---

Integrated in Hadoop-Common-trunk-Commit #1956 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1956/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242776#comment-13242776
 ] 

Hudson commented on MAPREDUCE-4082:
---

Integrated in Hadoop-Common-trunk-Commit #1956 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1956/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in 
 the module JAR
 

 Key: MAPREDUCE-4082
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4082.patch


 Currently the mrapp-generated-classpath file containing the 'built' 
 classpath, which only makes sense during building/testing in the machine 
 where the build happens, is bundled in the hadoop-mapreduce-client-app JAR.
 Because the file is bundled in the hadoop-mapreduce-client-app JAR, its 
 contents are added to the classpath of all MR jobs. 
 All this entries are useless and just pollute the classpath.
 This file should not be bundled in the hadoop-mapreduce-client-app JAR.
 As an example, the contents of this file in my local built are:
 {code}

[jira] [Updated] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters

2012-03-30 Thread Ahmed Radwan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4090:


  Component/s: mrv1
Affects Version/s: 0.23.1
   1.0.1

 Branch 1 pipes doesn't work on MR2 clusters
 ---

 Key: MAPREDUCE-4090
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1, mrv2
Affects Versions: 0.23.1, 1.0.1
Reporter: Ahmed Radwan

 If I compile pipes examples on branch 1: 
 {code}
 ant -Dcompile.c++=yes examples
 {code}
 And then try to run it on an MR2 cluster, the pipes job hangs forever at map 
 0% reduce 0%, I can see in the maps stderr:
 {code}
 Hadoop Pipes Exception: Illegal text protocol command 
 {code}
 The issue here is that if users have older pipes job, they won't be able to 
 run it on MR2 (without recompilation). Is this expected or there is something 
 to be fixed so jobs can be used interchangeably? Or should we document it as 
 an incompatibility?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4089:
---

Target Version/s: 0.23.2, 2.0.0, trunk  (was: trunk, 2.0.0, 0.23.2)
  Status: Open  (was: Patch Available)

 Hung Tasks never time out. 
 ---

 Key: MAPREDUCE-4089
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
 Attachments: MR-4089.txt


 The AM will timeout a task through mapreduce.task.timeout only when it does 
 not hear from the task within the given timeframe.  On 1.0 a task must be 
 making progress, either by reading input from HDFS, writing output to HDFS, 
 writing to a log, or calling a special method to inform it that it is still 
 making progress.
 This is because on 0.23 a status update which happens every 3 seconds is 
 counted as progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.

2012-03-30 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4089:
---

Attachment: MR-4089.txt

This patch addresses, a ping timeout, progress timeout, and a timeout of 0.

 Hung Tasks never time out. 
 ---

 Key: MAPREDUCE-4089
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2, 2.0.0, trunk
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Blocker
 Attachments: MR-4089.txt, MR-4089.txt


 The AM will timeout a task through mapreduce.task.timeout only when it does 
 not hear from the task within the given timeframe.  On 1.0 a task must be 
 making progress, either by reading input from HDFS, writing output to HDFS, 
 writing to a log, or calling a special method to inform it that it is still 
 making progress.
 This is because on 0.23 a status update which happens every 3 seconds is 
 counted as progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Attachment: MAPREDUCE-4092.patch

 commitJob Exception does not fail job (0.20 regression)
 ---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)

2012-03-30 Thread Jonathan Eagles (Created) (JIRA)

commitJob Exception does not fail job (0.20 regression)
---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch

If commitJob throws an exception JobImpl will swallow the exception with a 
warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob 
exception will fail the job

org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Status: Open  (was: Patch Available)

 commitJob Exception does not fail job (0.20 regression)
 ---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Attachment: MAPREDUCE-4092.patch

 commitJob Exception does not fail job (0.20 regression)
 ---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Status: Patch Available  (was: Open)

 commitJob Exception does not fail job (0.20 regression)
 ---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Status: Patch Available  (was: Open)

 commitJob Exception does not fail job (0.20 regression)
 ---

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Summary: commitJob Exception does not fail job (regression in 0.23 vs 0.20) 
 (was: commitJob Exception does not fail job (0.20 regression))

 commitJob Exception does not fail job (regression in 0.23 vs 0.20)
 --

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
 Job

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-4092:
---

Description: 
If commitJob throws an exception JobImpl will swallow the exception with a 
warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob 
exception will fail the job

Exception logged in the AM as WARN
  org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for 
Job
Job still finishes as succeeded

  was:
If commitJob throws an exception JobImpl will swallow the exception with a 
warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob 
exception will fail the job

org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for Job


 commitJob Exception does not fail job (regression in 0.23 vs 0.20)
 --

 Key: MAPREDUCE-4092
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Blocker
 Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch


 If commitJob throws an exception JobImpl will swallow the exception with a 
 warning and succeed the Job. This is a break from 0.20 and 1.0 where 
 commitJob exception will fail the job
 Exception logged in the AM as WARN
   org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit 
 for Job
 Job still finishes as succeeded

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242808#comment-13242808
 ] 

Hudson commented on MAPREDUCE-4091:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1969 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1969/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 tools testcases failing because of MAPREDUCE-4082
 -

 Key: MAPREDUCE-4091
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, test
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4091.patch


 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main 
 classpath to the test classpath.
 The objective of MAPREDUCE-4082 was to remove the generated-classpath file 
 from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the 
 test-classpath would do the trick.
 This is breaking tools testcases (most likely) because of different 
 classloader being used by maven for main classpath and test classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242809#comment-13242809
 ] 

Hudson commented on MAPREDUCE-4082:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1969 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1969/])
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) 
(Revision 1307623)

 Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1307623
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml


 hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in 
 the module JAR
 

 Key: MAPREDUCE-4082
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4082.patch


 Currently the mrapp-generated-classpath file containing the 'built' 
 classpath, which only makes sense during building/testing in the machine 
 where the build happens, is bundled in the hadoop-mapreduce-client-app JAR.
 Because the file is bundled in the hadoop-mapreduce-client-app JAR, its 
 contents are added to the classpath of all MR jobs. 
 All this entries are useless and just pollute the classpath.
 This file should not be bundled in the hadoop-mapreduce-client-app JAR.
 As an example, the contents of this file in my local built are:
 {code}

[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082

[
https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242827#comment-13242827
]

Hadoop QA commented on MAPREDUCE-4091:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520672/MAPREDUCE-4091.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapreduce.security.TestJHSSecurity

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2120//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2120//console

This message is automatically generated.

tools testcases failing because of MAPREDUCE-4082
-

Attachments: MAPREDUCE-4091.patch

[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.

[
https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242843#comment-13242843
]

Hadoop QA commented on MAPREDUCE-4089:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520679/MR-4089.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 508 javac compiler warnings (more
than the trunk's current 507 warnings).

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapreduce.security.TestJHSSecurity
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestClientRedirect

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2121//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2121//console

This message is automatically generated.

Hung Tasks never time out.
---

[jira] [Commented] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)