[jira] [Commented] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242844#comment-13242844 ] Hadoop QA commented on MAPREDUCE-4092: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520682/MAPREDUCE-4092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestJobSysDirWithDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2122//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2122//console This message is automatically generated. > commitJob Exception does not fail job (regression in 0.23 vs 0.20) > -- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > Exception logged in the AM as WARN > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit > for Job > Job still finishes as succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242843#comment-13242843 ] Hadoop QA commented on MAPREDUCE-4089: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520679/MR-4089.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 508 javac compiler warnings (more than the trunk's current 507 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.security.TestJHSSecurity org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestClientRedirect +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2121//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2121//console This message is automatically generated. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt, MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242827#comment-13242827 ] Hadoop QA commented on MAPREDUCE-4091: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520672/MAPREDUCE-4091.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2120//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2120//console This message is automatically generated. > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242809#comment-13242809 ] Hudson commented on MAPREDUCE-4082: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1969/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in > the module JAR > > > Key: MAPREDUCE-4082 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4082.patch > > > Currently the mrapp-generated-classpath file containing the 'built' > classpath, which only makes sense during building/testing in the machine > where the build happens, is bundled in the hadoop-mapreduce-client-app JAR. > Because the file is bundled in the hadoop-mapreduce-client-app JAR, its > contents are added to the classpath of all MR jobs. > All this entries are useless and just pollute the classpath. > This file should not be bundled in the hadoop-mapreduce-client-app JAR. > As an example, the contents of this file in my local built are: > {code} > /Users/tucu/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/tucu/.m2/repository/asm/asm/3.2/asm-3.2.jar:/Users/tucu/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar:/Users/tucu/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/tucu/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/Users/tucu/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/Users/tucu/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/Users/tucu/.m2/repository/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar:/Users/tucu/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-client/1.8/jersey-client-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-grizzly2/1.8/jersey-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.8/jersey-guice-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-core/1.8/jersey-test-framework-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-grizzly2/1.8/jersey-test-framework-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/tucu/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/tucu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/tucu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/tucu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/tucu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/tucu/.m2/repository/commons-daemon/commons-daemon/1.0.3/commons-daemon-1.0.3.jar:/Users/tucu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/tucu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/tucu/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/tucu/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/tucu/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar:/Users/tucu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/tucu/.m2/repository/hsqldb/hsqldb/1.8.0.7/hsqldb-1.8.0.7.jar:/Users/tucu/.m2
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242808#comment-13242808 ] Hudson commented on MAPREDUCE-4091: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1969/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Description: If commitJob throws an exception JobImpl will swallow the exception with a warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob exception will fail the job Exception logged in the AM as WARN org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for Job Job still finishes as succeeded was: If commitJob throws an exception JobImpl will swallow the exception with a warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob exception will fail the job org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for Job > commitJob Exception does not fail job (regression in 0.23 vs 0.20) > -- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > Exception logged in the AM as WARN > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit > for Job > Job still finishes as succeeded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (regression in 0.23 vs 0.20)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Summary: commitJob Exception does not fail job (regression in 0.23 vs 0.20) (was: commitJob Exception does not fail job (0.20 regression)) > commitJob Exception does not fail job (regression in 0.23 vs 0.20) > -- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Status: Patch Available (was: Open) > commitJob Exception does not fail job (0.20 regression) > --- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Status: Patch Available (was: Open) > commitJob Exception does not fail job (0.20 regression) > --- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Status: Open (was: Patch Available) > commitJob Exception does not fail job (0.20 regression) > --- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Attachment: MAPREDUCE-4092.patch > commitJob Exception does not fail job (0.20 regression) > --- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch, MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4092: --- Attachment: MAPREDUCE-4092.patch > commitJob Exception does not fail job (0.20 regression) > --- > > Key: MAPREDUCE-4092 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Blocker > Attachments: MAPREDUCE-4092.patch > > > If commitJob throws an exception JobImpl will swallow the exception with a > warning and succeed the Job. This is a break from 0.20 and 1.0 where > commitJob exception will fail the job > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for > Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4092) commitJob Exception does not fail job (0.20 regression)
commitJob Exception does not fail job (0.20 regression) --- Key: MAPREDUCE-4092 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4092 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-4092.patch If commitJob throws an exception JobImpl will swallow the exception with a warning and succeed the Job. This is a break from 0.20 and 1.0 where commitJob exception will fail the job org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Could not do commit for Job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4089: --- Attachment: MR-4089.txt This patch addresses, a ping timeout, progress timeout, and a timeout of 0. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt, MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4089: --- Target Version/s: 0.23.2, 2.0.0, trunk (was: trunk, 2.0.0, 0.23.2) Status: Patch Available (was: Open) > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt, MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4089: --- Target Version/s: 0.23.2, 2.0.0, trunk (was: trunk, 2.0.0, 0.23.2) Status: Open (was: Patch Available) > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-4090: Component/s: mrv1 Affects Version/s: 0.23.1 1.0.1 > Branch 1 pipes doesn't work on MR2 clusters > --- > > Key: MAPREDUCE-4090 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 0.23.1, 1.0.1 >Reporter: Ahmed Radwan > > If I compile pipes examples on branch 1: > {code} > ant -Dcompile.c++=yes examples > {code} > And then try to run it on an MR2 cluster, the pipes job hangs forever at map > 0% reduce 0%, I can see in the maps stderr: > {code} > Hadoop Pipes Exception: Illegal text protocol command > {code} > The issue here is that if users have older pipes job, they won't be able to > run it on MR2 (without recompilation). Is this expected or there is something > to be fixed so jobs can be used interchangeably? Or should we document it as > an incompatibility? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242776#comment-13242776 ] Hudson commented on MAPREDUCE-4082: --- Integrated in Hadoop-Common-trunk-Commit #1956 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1956/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in > the module JAR > > > Key: MAPREDUCE-4082 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4082.patch > > > Currently the mrapp-generated-classpath file containing the 'built' > classpath, which only makes sense during building/testing in the machine > where the build happens, is bundled in the hadoop-mapreduce-client-app JAR. > Because the file is bundled in the hadoop-mapreduce-client-app JAR, its > contents are added to the classpath of all MR jobs. > All this entries are useless and just pollute the classpath. > This file should not be bundled in the hadoop-mapreduce-client-app JAR. > As an example, the contents of this file in my local built are: > {code} > /Users/tucu/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/tucu/.m2/repository/asm/asm/3.2/asm-3.2.jar:/Users/tucu/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar:/Users/tucu/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/tucu/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/Users/tucu/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/Users/tucu/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/Users/tucu/.m2/repository/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar:/Users/tucu/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-client/1.8/jersey-client-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-grizzly2/1.8/jersey-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.8/jersey-guice-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-core/1.8/jersey-test-framework-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-grizzly2/1.8/jersey-test-framework-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/tucu/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/tucu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/tucu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/tucu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/tucu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/tucu/.m2/repository/commons-daemon/commons-daemon/1.0.3/commons-daemon-1.0.3.jar:/Users/tucu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/tucu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/tucu/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/tucu/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/tucu/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar:/Users/tucu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/tucu/.m2/repository/hsqldb/hsqldb/1.8.0.7/hsqldb-1.8.0.7.jar:/Users/tucu/.m2/repos
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242775#comment-13242775 ] Hudson commented on MAPREDUCE-4091: --- Integrated in Hadoop-Common-trunk-Commit #1956 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1956/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242771#comment-13242771 ] Hudson commented on MAPREDUCE-4091: --- Integrated in Hadoop-Hdfs-trunk-Commit #2031 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2031/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4067) Replace YarnRemoteException with IOException in MRv2 APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242773#comment-13242773 ] Siddharth Seth commented on MAPREDUCE-4067: --- YarnRemoteException was serializing the actual exception and sending it over the wire. It was missing an unwind method to get the actual exception. IAC - clients should not be expected to unwind and that should've been handled in the RPC layer itself. Instead of having the APIs throw an IOException, I'd prefer to define a Yarn / HadoopException which extends Exception and can later be sub-classed for more specific exceptions. Also, possibly add error codes to these exceptions as well as to what is sent over the wire. That may make things easier for clients written in other languages. > Replace YarnRemoteException with IOException in MRv2 APIs > - > > Key: MAPREDUCE-4067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4067 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Siddharth Seth >Priority: Blocker > > YarnRemoteException is defined as a generic wrapper for all the exceptions in > yarn. I think we should instead throw IOExceptions in the API, which can > later be extended for more specialized exceptions without breaking > compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4082) hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242772#comment-13242772 ] Hudson commented on MAPREDUCE-4082: --- Integrated in Hadoop-Hdfs-trunk-Commit #2031 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2031/]) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) (Revision 1307623) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307623 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml > hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in > the module JAR > > > Key: MAPREDUCE-4082 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4082 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4082.patch > > > Currently the mrapp-generated-classpath file containing the 'built' > classpath, which only makes sense during building/testing in the machine > where the build happens, is bundled in the hadoop-mapreduce-client-app JAR. > Because the file is bundled in the hadoop-mapreduce-client-app JAR, its > contents are added to the classpath of all MR jobs. > All this entries are useless and just pollute the classpath. > This file should not be bundled in the hadoop-mapreduce-client-app JAR. > As an example, the contents of this file in my local built are: > {code} > /Users/tucu/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/tucu/.m2/repository/asm/asm/3.2/asm-3.2.jar:/Users/tucu/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar:/Users/tucu/.m2/repository/com/google/guava/guava/r09/guava-r09.jar:/Users/tucu/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/Users/tucu/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/Users/tucu/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/Users/tucu/.m2/repository/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar:/Users/tucu/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-client/1.8/jersey-client-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-grizzly2/1.8/jersey-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.8/jersey-guice-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-core/1.8/jersey-test-framework-core-1.8.jar:/Users/tucu/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-grizzly2/1.8/jersey-test-framework-grizzly2-1.8.jar:/Users/tucu/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/tucu/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/tucu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/tucu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/tucu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/tucu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/tucu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/tucu/.m2/repository/commons-daemon/commons-daemon/1.0.3/commons-daemon-1.0.3.jar:/Users/tucu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/tucu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/tucu/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/tucu/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/Users/tucu/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/tucu/.m2/repository/commons-logging/commons-logging-api/1.1/commons-logging-api-1.1.jar:/Users/tucu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/Users/tucu/.m2/repository/hsqldb/hsqldb/1.8.0.7/hsqldb-1.8.0.7.jar:/Users/tucu/.m2/repositor
[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4091: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk & branch-2 > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242760#comment-13242760 ] Mahadev konar commented on MAPREDUCE-4091: -- +1 for the patch. Looks good. > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4091: -- Status: Patch Available (was: Open) > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4062) AM Launcher thread can hang forever
[ https://issues.apache.org/jira/browse/MAPREDUCE-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242754#comment-13242754 ] Jason Lowe commented on MAPREDUCE-4062: --- We've seen an issue where using both an RPC-level timeout (in this case the ping timeout) and the timer task can cause the AM to lose track of a container and hang the job. Here's the relevant part of the AM log: {noformat} 2012-03-29 07:32:17,794 ERROR [ContainerLauncher #199] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:container_1333003059741_0010_01_003408 (auth:SIMPLE) cause:java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:x remote=x.x.x.x.x/x.x.x.x:x] 2012-03-29 07:32:17,794 WARN [ContainerLauncher #199] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:x remote=x.x.x.x.x/x.x.x.x:x] 2012-03-29 07:32:17,794 ERROR [ContainerLauncher #199] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:container_1333003059741_0010_01_003408 (auth:SIMPLE) cause:java.io.IOException: java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:x remote=x.x.x.x.x/x.x.x.x:x] 2012-03-29 07:32:17,795 WARN [Timer-1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Couldn't complete CONTAINER_REMOTE_CLEANUP on container_1333003059741_0010_01_003408/attempt_1333003059741_0010_m_003097_0. Interrupting and returning 2012-03-29 07:32:17,798 INFO [Timer-1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Stack trace of the command-thread: at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuilder.append(StringBuilder.java:119) at java.lang.StackTraceElement.toString(StackTraceElement.java:157) at java.lang.String.valueOf(String.java:2826) at java.lang.StringBuilder.append(StringBuilder.java:115) at java.lang.Throwable.printStackTrace(Throwable.java:512) at org.apache.hadoop.util.StringUtils.stringifyException(StringUtils.java:64) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:260) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:479) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-03-29 07:32:17,800 WARN [ContainerLauncher #199] org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294) at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:179) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:263) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:479) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) {noformat} Looks like the socket timeout and the timer task timeout occurred almost simultaneously. The socket exception was caught first, and during the catch clause we fielded the interrupted exception. That broke us out of the handling of the socket exception and we never marked the container status properly before leaving. So I'm +1 on removing the timer task and relying on the rpcTimeout, unless there are other cases besides RPC where we'd expect the container launcher to get stuck. > AM Launcher thread can hang forever > --- > > Key: MAPREDUCE-4062 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4062 > Project: Hadoop Map/Reduce
[jira] [Updated] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4091: -- Attachment: MAPREDUCE-4091.patch the tools (streaming & archives) failing testcases are passing again. and the hadoop-mapreduce-client-app JAR does not have the mrapp-generated-classpath file in it. > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4091.patch > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
[ https://issues.apache.org/jira/browse/MAPREDUCE-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242749#comment-13242749 ] Alejandro Abdelnur commented on MAPREDUCE-4091: --- The generated-classpath file has a meaning for running testcases in the current project. If downstream projects want to leverage that file, they have create it in their project (i.e. oozie-core creates one to be able to use MiniMR). Having the generated-classpath file bundled in the JAR is pointless, and as MAPREDUCE-4082 stated, pollutes the classpath big time. An alternate way to achieve not having the generated-classpath file in the JAR is to excluded it in the maven-jar-plugin. > tools testcases failing because of MAPREDUCE-4082 > - > > Key: MAPREDUCE-4091 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > > MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main > classpath to the test classpath. > The objective of MAPREDUCE-4082 was to remove the generated-classpath file > from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the > test-classpath would do the trick. > This is breaking tools testcases (most likely) because of different > classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4091) tools testcases failing because of MAPREDUCE-4082
tools testcases failing because of MAPREDUCE-4082 - Key: MAPREDUCE-4091 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4091 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, test Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Priority: Critical Fix For: 2.0.0 MAPREDUCE-4082 moved the generated-classpath file used by MRApp from the main classpath to the test classpath. The objective of MAPREDUCE-4082 was to remove the generated-classpath file from the hadoop-mapreduce-client-app JAR. I've thought that moving it to the test-classpath would do the trick. This is breaking tools testcases (most likely) because of different classloader being used by maven for main classpath and test classpath. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242705#comment-13242705 ] Hadoop QA commented on MAPREDUCE-4089: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520653/MR-4089.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2119//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2119//console This message is automatically generated. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4090) Branch 1 pipes doesn't work on MR2 clusters
Branch 1 pipes doesn't work on MR2 clusters --- Key: MAPREDUCE-4090 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4090 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Ahmed Radwan If I compile pipes examples on branch 1: {code} ant -Dcompile.c++=yes examples {code} And then try to run it on an MR2 cluster, the pipes job hangs forever at map 0% reduce 0%, I can see in the maps stderr: {code} Hadoop Pipes Exception: Illegal text protocol command {code} The issue here is that if users have older pipes job, they won't be able to run it on MR2 (without recompilation). Is this expected or there is something to be fixed so jobs can be used interchangeably? Or should we document it as an incompatibility? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242675#comment-13242675 ] Robert Joseph Evans commented on MAPREDUCE-4089: OK Talking to people here it looks like there are a significant number of users that I know of that set the timeout to 0, so I am going to come up with a new patch to have the 0 timeout be acceptable too. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242671#comment-13242671 ] Kihwal Lee commented on MAPREDUCE-4072: --- Users should not include -Djava.library.path in task jvm options. The code change looks good, but we need to make sure this is documented somewhere as an incompatible change. > User set java.library.path seems to overwrite default creating problems > native lib loading > -- > > Key: MAPREDUCE-4072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Anupam Seth >Assignee: Anupam Seth > Attachments: MAPREDUCE-4072-branch-23.patch > > > This was found by Peeyush Bishnoi. > While running a distributed cache example with Hadoop-0.23, > tasks are failing as follows: > > Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: at > org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at > org.apache.hadoop.util.Shell.run(Shell.java:188) at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at > java.util.concurrent.FutureTask.run(FutureTask.java:138) at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user > is > > Same Pig script and command work successfully on 0.20 > See this in the stderr: > Exception in thread "main" java.lang.ExceptionInInitializerError > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) > at org.apache.hadoop.security.Groups.(Groups.java:54) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) > at > org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) > Caused by: java.lang.RuntimeException: Bailing out since native library > couldn't be loaded > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) > ... 12 more > Pig command: > $ pig -Dmapred.job.queue.name= -Dmapred.cache.archives= > -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib > -Dip2geo.preLoadLibraries=" -Djava.io.tmpdir=/grid/0/tmp > -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4089: --- Attachment: MR-4089.txt This patch addresses the timeout issue, and it does so by making ping not update progress. It is still not completely compatible with 1.0, as in 1.0 if the timeout is set to 0 the task will never timeout. But because this patch makes it so ping is ignored a task that has a timeout of 0, but is so locked up that it cannot ping anymore will never timeout. I am planning to address these in a follow on JIRA, unless someone has some objections to doing so. I also have not run all of the unit tests yet. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4089: --- Target Version/s: 0.23.2, 2.0.0, trunk (was: trunk, 2.0.0, 0.23.2) Status: Patch Available (was: Open) > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4089.txt > > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242657#comment-13242657 ] Hadoop QA commented on MAPREDUCE-4072: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520642/MAPREDUCE-4072-branch-23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2118//console This message is automatically generated. > User set java.library.path seems to overwrite default creating problems > native lib loading > -- > > Key: MAPREDUCE-4072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Anupam Seth >Assignee: Anupam Seth > Attachments: MAPREDUCE-4072-branch-23.patch > > > This was found by Peeyush Bishnoi. > While running a distributed cache example with Hadoop-0.23, > tasks are failing as follows: > > Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: at > org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at > org.apache.hadoop.util.Shell.run(Shell.java:188) at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at > java.util.concurrent.FutureTask.run(FutureTask.java:138) at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user > is > > Same Pig script and command work successfully on 0.20 > See this in the stderr: > Exception in thread "main" java.lang.ExceptionInInitializerError > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) > at org.apache.hadoop.security.Groups.(Groups.java:54) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) > at > org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) > Caused by: java.lang.RuntimeException: Bailing out since native library > couldn't be loaded > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) > ... 12 more > Pig command: > $ pig -Dmapred.job.queue.name= -Dmapred.cache.archives= > -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib > -Dip2geo.preLoadLibraries=" -Djava.io.tmpdir=/grid/0/tm
[jira] [Updated] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4059: --- Status: Open (was: Patch Available) Addressing review comments. > The history server should have a separate pluggable storage/query interface > --- > > Key: MAPREDUCE-4059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4059.txt, MR-4059.txt > > > The history server currently caches all parsed jobs in RAM. These jobs can > be very large because of counters. It would be nice to have a pluggable > interface for the cacheing and querying of the cached data so that we can > play around with different implementations. Also just for cleanness of the > code it would be nice to split the very large JobHistoryServer.java into a > few smaller ones that are more understandable and readable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242635#comment-13242635 ] Robert Joseph Evans commented on MAPREDUCE-4059: Oh I should also remove my debugging log messages that are logged at error!! > The history server should have a separate pluggable storage/query interface > --- > > Key: MAPREDUCE-4059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4059.txt, MR-4059.txt > > > The history server currently caches all parsed jobs in RAM. These jobs can > be very large because of counters. It would be nice to have a pluggable > interface for the cacheing and querying of the cached data so that we can > play around with different implementations. Also just for cleanness of the > code it would be nice to split the very large JobHistoryServer.java into a > few smaller ones that are more understandable and readable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242632#comment-13242632 ] Robert Joseph Evans commented on MAPREDUCE-4059: That is a great catch. I did totally miss the possibility of int overflow, and with count being negative. Will update the patch. Thanks. > The history server should have a separate pluggable storage/query interface > --- > > Key: MAPREDUCE-4059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4059.txt, MR-4059.txt > > > The history server currently caches all parsed jobs in RAM. These jobs can > be very large because of counters. It would be nice to have a pluggable > interface for the cacheing and querying of the cached data so that we can > play around with different implementations. Also just for cleanness of the > code it would be nice to split the very large JobHistoryServer.java into a > few smaller ones that are more understandable and readable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4089) Hung Tasks never time out.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242620#comment-13242620 ] Robert Joseph Evans commented on MAPREDUCE-4089: >From looking at the code for Task. When TaskReporter.progress() is called, >all it does is set the progress flag, which causes a progress update to be >sent to the AM instead of a ping to be sent. So I would guess is that we want >to stop counting pings when measuring progress in the AM. So it should >probably be a simple 1 line change. > Hung Tasks never time out. > --- > > Key: MAPREDUCE-4089 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2, 2.0.0, trunk >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > > The AM will timeout a task through mapreduce.task.timeout only when it does > not hear from the task within the given timeframe. On 1.0 a task must be > making progress, either by reading input from HDFS, writing output to HDFS, > writing to a log, or calling a special method to inform it that it is still > making progress. > This is because on 0.23 a status update which happens every 3 seconds is > counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4072: --- Attachment: MAPREDUCE-4072-branch-23.patch Uploading patch to remove transmission of java.library.path to container. > User set java.library.path seems to overwrite default creating problems > native lib loading > -- > > Key: MAPREDUCE-4072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Anupam Seth >Assignee: Anupam Seth > Attachments: MAPREDUCE-4072-branch-23.patch > > > This was found by Peeyush Bishnoi. > While running a distributed cache example with Hadoop-0.23, > tasks are failing as follows: > > Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: at > org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at > org.apache.hadoop.util.Shell.run(Shell.java:188) at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at > java.util.concurrent.FutureTask.run(FutureTask.java:138) at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user > is > > Same Pig script and command work successfully on 0.20 > See this in the stderr: > Exception in thread "main" java.lang.ExceptionInInitializerError > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) > at org.apache.hadoop.security.Groups.(Groups.java:54) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) > at > org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) > Caused by: java.lang.RuntimeException: Bailing out since native library > couldn't be loaded > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) > ... 12 more > Pig command: > $ pig -Dmapred.job.queue.name= -Dmapred.cache.archives= > -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib > -Dip2geo.preLoadLibraries=" -Djava.io.tmpdir=/grid/0/tmp > -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4072) User set java.library.path seems to overwrite default creating problems native lib loading
[ https://issues.apache.org/jira/browse/MAPREDUCE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-4072: --- Status: Patch Available (was: Open) > User set java.library.path seems to overwrite default creating problems > native lib loading > -- > > Key: MAPREDUCE-4072 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4072 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Anupam Seth >Assignee: Anupam Seth > Attachments: MAPREDUCE-4072-branch-23.patch > > > This was found by Peeyush Bishnoi. > While running a distributed cache example with Hadoop-0.23, > tasks are failing as follows: > > Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: at > org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at > org.apache.hadoop.util.Shell.run(Shell.java:188) at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:207) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:241) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at > java.util.concurrent.FutureTask.run(FutureTask.java:138) at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) main : command provided 1 main : user > is > > Same Pig script and command work successfully on 0.20 > See this in the stderr: > Exception in thread "main" java.lang.ExceptionInInitializerError > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1179) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1149) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1238) > at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264) > at org.apache.hadoop.security.Groups.(Groups.java:54) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:178) > at > org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:265) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:75) > Caused by: java.lang.RuntimeException: Bailing out since native library > couldn't be loaded > at > org.apache.hadoop.security.JniBasedUnixGroupsMapping.(JniBasedUnixGroupsMapping.java:48) > ... 12 more > Pig command: > $ pig -Dmapred.job.queue.name= -Dmapred.cache.archives= > -Dmapred.child.java.opts="-Djava.library.path=./ygeo/lib > -Dip2geo.preLoadLibraries=" -Djava.io.tmpdir=/grid/0/tmp > -Dmapred.create.symlink=yes -Dmapred.job.map.memory.mb=3072 piggeoscript.pig -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4089) Hung Tasks never time out.
Hung Tasks never time out. --- Key: MAPREDUCE-4089 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4089 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2, 2.0.0, trunk Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker The AM will timeout a task through mapreduce.task.timeout only when it does not hear from the task within the given timeframe. On 1.0 a task must be making progress, either by reading input from HDFS, writing output to HDFS, writing to a log, or calling a special method to inform it that it is still making progress. This is because on 0.23 a status update which happens every 3 seconds is counted as progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have a separate pluggable storage/query interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242600#comment-13242600 ] Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059: --- Hi Robert, I just have gone through a portion of the patch and have few comments/doubts on the patch. 1. {code:title=CachedHistoryStorage.java|borderStyle=solid} if(offset == null || offset < 0) offset = 0l; if(count == null) count = Long.MAX_VALUE; long at = 0; long end = offset + count - 1; LOG.error("Looking for entries starting at "+offset+" with a length of "+count+" so ending at "+end); for (Job job : jobs) { LOG.error("Looking at job END: "+at+" <= "+end); if(at > end) { break; } {code} Suppose say, in the above code offset is set as 100 and count is not initialized, so count will be initialized as Long.MAX_VALUE. If so, *end* may get negative value (because of integer overflow) and hence for loop will be exited in the first iteration itself. But I think, we should return entries starting from 100th entry. Am I missing anything here? 2. I think it is better to add sanity check like this in CachedHistoryStorage#getPartialJobs() {noformat} if(offset > jobs.size()) { return allJobs; } {noformat} > The history server should have a separate pluggable storage/query interface > --- > > Key: MAPREDUCE-4059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4059.txt, MR-4059.txt > > > The history server currently caches all parsed jobs in RAM. These jobs can > be very large because of counters. It would be nice to have a pluggable > interface for the cacheing and querying of the cached data so that we can > play around with different implementations. Also just for cleanness of the > code it would be nice to split the very large JobHistoryServer.java into a > few smaller ones that are more understandable and readable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing
Task stuck in JobLocalizer prevented other tasks on the same node from committing - Key: MAPREDUCE-4088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ravi Prakash Priority: Critical We saw that as a result of HADOOP-6963, one task was stuck in this Thread 23668: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) TONS MORE OF THIS SAME LINE - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) . . - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Interpreted frame) ne=451 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration, java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) @bci=150, line=324 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration) @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, java.lang.String, org.apache.hadoop.fs.Path, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, line=530 (Interpreted frame) While all other tasks on the same node were stuck in Thread 32141: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted frame) - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 (Interpreted frame) This should never happen. A stuck task should never prevent other tasks from different jobs on the same node from committing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Attachment: MR-4079-trunk.txt MR-4079-full-branch-0.23.txt The trunk patch depends on the patch in HADOOP-8227. I will put this JIRA in patch available once it is integrated in. Unfortunetly for 0.23 the RPC engine code is split among the projects so the patches for MAPREDUCE-4079 and HADOOP-8227 need to go in at the same time. I have attached the full patch for branch-0.23 for both of these JIRAs. Once both JIRAs are approved for trunk and branch-2.0, I can put the full patch into branch-0.23. > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4085) Kill task attempts longer than a configured queue max time
[ https://issues.apache.org/jira/browse/MAPREDUCE-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242427#comment-13242427 ] Allen Wittenauer commented on MAPREDUCE-4085: - It seems very wrong to me to include this functionality in a specific scheduler. The other thing to keep in mind is that at least in our use case, we want to punish jobs in that are essentially in the wrong queue. Pre-emption works out to be the incorrect action in this case; we ultimately do want to the job to be failed as negative feedback to the user that they did something wrong, especially for primarily ad hoc usage (or, as you said, very heterogeneous). > Kill task attempts longer than a configured queue max time > -- > > Key: MAPREDUCE-4085 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4085 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: task >Reporter: Allen Wittenauer > > For some environments, it is desirable to have certain queues have an SLA > with regards to task turnover. (i.e., a slot will be free in X minutes and > scheduled to the appropriate job) Queues should have a 'task time limit' > that would cause task attempts over this time to be killed. This leaves open > the possibility that if the task was on a bad node, it could still be > rescheduled up to max.task.attempt times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4085) Kill task attempts longer than a configured queue max time
[ https://issues.apache.org/jira/browse/MAPREDUCE-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242391#comment-13242391 ] Robert Joseph Evans commented on MAPREDUCE-4085: I can see the need for something like this, to ensure that new jobs can run and meet their SLAs, but I think it would be better to have it be part of a preemption like mechanism, where we let the tasks run until there is some other Task/Container(for MRv2) that is requested. Once there is a need for those resources iff the current task/container has gone over the configured limit the JT/RM, on the next heartbeat, can inform the TT/NM to kill the task/container. The fair scheduler already supports preemption and perhaps this could be added there. MAPREDUCE-3938 was filed to add preemption to the Capacity Scheduler for 2.0 and it might be good to add this in as part of the design there. I don't really like the idea of having a hard limit on the runtime. What is more if there is a hard limit on how long a task can run for I see very little benefit in having it rescheduled more then once. If it was a slow node, then OK we can pick another node and it might finish in time, but unless the cluster is very heterogeneous the task is just going to run to the maximum time limit 4 times and then the Job will be failed. > Kill task attempts longer than a configured queue max time > -- > > Key: MAPREDUCE-4085 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4085 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: task >Reporter: Allen Wittenauer > > For some environments, it is desirable to have certain queues have an SLA > with regards to task turnover. (i.e., a slot will be free in X minutes and > scheduled to the appropriate job) Queues should have a 'task time limit' > that would cause task attempts over this time to be killed. This leaves open > the possibility that if the task was on a bad node, it could still be > rescheduled up to max.task.attempt times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242375#comment-13242375 ] Hudson commented on MAPREDUCE-3916: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1967 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1967/]) MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with 0.20.203.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242354#comment-13242354 ] Hudson commented on MAPREDUCE-3377: --- Integrated in Hadoop-Mapreduce-trunk #1035 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/]) MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java > Compatibility issue with 0.20.203. > -- > > Key: MAPREDUCE-3377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.203.0 > Environment: 0.20.203.0 standalone JDK 1.6 Linux >Reporter: Jane Chen >Assignee: Jane Chen > Fix For: 1.0.3, 2.0.0 > > Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, > mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, > mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before > > > I have an OutputFormat which implements Configurable. I set new config > entries to a job configuration during checkOutputSpec() so that the tasks > will get the config entries through the job configuration. This works fine > in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my > OutputFormat still has the configuration set, but the copy a task gets does > not have the new entries that are set as part of checkOutputSpec(). > I believe that the problem is with JobClient. The job configuration needs to > wait till checkOutputSpec() is returned before being cloned and submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4081) TestMROutputFormat.java does not compile
[ https://issues.apache.org/jira/browse/MAPREDUCE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242357#comment-13242357 ] Hudson commented on MAPREDUCE-4081: --- Integrated in Hadoop-Mapreduce-trunk #1035 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/]) MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via bobby) (Revision 1306901) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306901 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java > TestMROutputFormat.java does not compile > > > Key: MAPREDUCE-4081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4081 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, mrv2 >Affects Versions: 2.0.0, trunk >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4081.patch > > > [ERROR] > /hadoop/src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java:[36,7] > class TestConfInCheckSpec is public, should be declared in a file named > TestConfInCheckSpec.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242342#comment-13242342 ] Hudson commented on MAPREDUCE-3989: --- Integrated in Hadoop-Mapreduce-trunk #1035 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/]) MAPREDUCE-3989. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307102) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307102 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242338#comment-13242338 ] Hudson commented on MAPREDUCE-3916: --- Integrated in Hadoop-Common-trunk-Commit #1954 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1954/]) MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242337#comment-13242337 ] Hudson commented on MAPREDUCE-3916: --- Integrated in Hadoop-Hdfs-trunk-Commit #2029 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2029/]) MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) (Revision 1307413) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307413 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/proxy/.keep * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242335#comment-13242335 ] Devaraj K commented on MAPREDUCE-3916: -- Thanks a lot Alejandro for reviewing and verifying the patch. {quote} -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {quote} No new tests required because changes done in yarn-default.xml and added one dir. > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3916: -- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Devaraj. I had to manually create the proxy/.keep file as the patch command ignored it. Committed to trunk and branch-2 > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242329#comment-13242329 ] Hadoop QA commented on MAPREDUCE-3916: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520600/MAPREDUCE-3916.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2117//console This message is automatically generated. > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.j
[jira] [Commented] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242328#comment-13242328 ] Alejandro Abdelnur commented on MAPREDUCE-3916: --- +1. built/deployed pseudo cluster, set yarn.web-proxy.address and proxy started successfully > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4081) TestMROutputFormat.java does not compile
[ https://issues.apache.org/jira/browse/MAPREDUCE-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242319#comment-13242319 ] Hudson commented on MAPREDUCE-4081: --- Integrated in Hadoop-Hdfs-trunk #1000 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/]) MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via bobby) (Revision 1306901) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306901 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java > TestMROutputFormat.java does not compile > > > Key: MAPREDUCE-4081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4081 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, mrv2 >Affects Versions: 2.0.0, trunk >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4081.patch > > > [ERROR] > /hadoop/src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java:[36,7] > class TestConfInCheckSpec is public, should be declared in a file named > TestConfInCheckSpec.java -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3989) cap space usage of default log4j rolling policy (mr specific changes)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242313#comment-13242313 ] Hudson commented on MAPREDUCE-3989: --- Integrated in Hadoop-Hdfs-trunk #1000 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/]) MAPREDUCE-3989. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307102) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307102 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh > cap space usage of default log4j rolling policy (mr specific changes) > - > > Key: MAPREDUCE-3989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3989.patch > > > see HADOOP-8149 for background on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3916: - Priority: Critical (was: Major) Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K >Priority: Critical > Labels: mrv2 > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3916: - Labels: mrv2 (was: ) Target Version/s: 2.0.0, 3.0.0 Affects Version/s: 3.0.0 2.0.0 Status: Patch Available (was: Open) {quote}Setting yarn.web-proxy.address to match yarn.resourcemanager.address doesn't disable the proxyserver (althought not setting yarn.web-proxy.address at all correctly disable it and produces a message: org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run). This contradicts the documentation provided for yarn.web-proxy.address in yarn-default.xml{quote} If we don't set 'yarn.web-proxy.address', then the proxy server will not start as separate process and start up fails with the message 'org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run' and proxy will run as part of the RM. If we give the value for 'yarn.web-proxy.address' which matches to the 'yarn.resourcemanager.address' then proxy server start up fails with "Address already in use". And also it is not starting as part of the RM. It is not as per the documentation. I have modified the description for this property accordingly. {quote} Setting yarn.web-proxy.address and running the service results in the following: {quote} I have added the 'hadoop-yarn-common\src\main\resources\webapps\proxy' dir with .keep file in it to resolve this problem as done for other components. > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1, 2.0.0, 3.0.0 >Reporter: Roman Shaposhnik >Assignee: Devaraj K > Labels: mrv2 > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.ap
[jira] [Assigned] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-3916: Assignee: Devaraj K > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Devaraj K > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3916) various issues with running yarn proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3916: - Attachment: MAPREDUCE-3916.patch > various issues with running yarn proxyserver > > > Key: MAPREDUCE-3916 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager, webapps >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Devaraj K > Attachments: MAPREDUCE-3916.patch > > > Seem like yarn proxyserver is not operational when running out of the 0.23.1 > RC2 tarball. > # Setting yarn.web-proxy.address to match yarn.resourcemanager.address > doesn't disable the proxyserver (althought not setting yarn.web-proxy.address > at all correctly disable it and produces a message: > org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so > the proxy will not run). This contradicts the documentation provided for > yarn.web-proxy.address in yarn-default.xml > # Setting yarn.web-proxy.address and running the service results in the > following: > {noformat} > $ ./sbin/yarn-daemon.sh start proxyserver > starting proxyserver, logging to > /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out > /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m > -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= > -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native > -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs > -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs > -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log > -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA > -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath > /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > {noformat} > with the following message found in the logs: > {noformat} > 2012-02-24 09:26:31,099 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at > org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4037) Fails to start proxy server due to webapps/proxy not found in CLASSPATH
[ https://issues.apache.org/jira/browse/MAPREDUCE-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K resolved MAPREDUCE-4037. -- Resolution: Duplicate Assignee: (was: Devaraj K) Duplicate of MAPREDUCE-3916. > Fails to start proxy server due to webapps/proxy not found in CLASSPATH > --- > > Key: MAPREDUCE-4037 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4037 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Devaraj K >Priority: Critical > > {code:xml} > 2012-03-20 10:41:35,688 FATAL > org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web > server > java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:90) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > 2012-03-20 10:41:35,689 ERROR > org.apache.hadoop.yarn.service.CompositeService: Error starting services > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer > org.apache.hadoop.yarn.YarnException: Could not start proxy web server > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:100) > at > org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) > Caused by: java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH > at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) > at > org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:90) > ... 2 more > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242285#comment-13242285 ] Hadoop QA commented on MAPREDUCE-4087: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520595/4087.trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2116//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2116//console This message is automatically generated. > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch, 4087.trunk.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4033) MiniMRClientClusterFactory is not setting the temp dir correctly in the conf used to init MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved MAPREDUCE-4033. --- Resolution: Invalid Assignee: Alejandro Abdelnur Oozie testcases are not using the MiniMR configuration to submit a job, thus missing settings shared between MiniMR and the AMs. Still it seems to be an issue when the staging HDFS directory of the AM and the JH differ and the app never ends. will verify and open JIRA if appropriate. > MiniMRClientClusterFactory is not setting the temp dir correctly in the conf > used to init MiniMRYarnCluster > --- > > Key: MAPREDUCE-4033 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4033 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 2.0.0 > > Attachments: minicluster-oozie-pig.txt > > > Oozie testcases are failing randomly because MR2 reports the job as unknown. > This seems to happen when Oozie queries via JobClient.getJob() for a > that just finished. > {code} > org.apache.oozie.action.ActionExecutorException: JA017: Unknown hadoop job > [job_1332176678205_0011] associated with action > [000-120319101023910-oozie-tucu-W@pig-action]. Failing this action! > {code} > Oozie reports this error when JobClient.getJob() returns NULL. > Looking at the mini cluster logs the job definitely run. > {code} > find . -name "*1332176678205_0011*" > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_02 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1332176678205_0011/container_1332176678205_0011_01_01 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_02 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1332176678205_0011/container_1332176678205_0011_01_01 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_02 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1332176678205_0011/container_1332176678205_0011_01_01 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_02 > ./core/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_1/application_1332176678205_0011/container_1332176678205_0011_01_01 > {code} > It seems there is a gap until the the job is avail in the JH server. > If this gap is unavoidable we need to ensure Oozie always waits at least the > gap time before querying for a job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242275#comment-13242275 ] Hadoop QA commented on MAPREDUCE-4087: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520592/4087.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2115//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2115//console This message is automatically generated. > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch, 4087.trunk.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4087: Hadoop Flags: Reviewed Status: Patch Available (was: Open) > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch, 4087.trunk.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4087: Attachment: 4087.trunk.patch Attaching patch for trunk. > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch, 4087.trunk.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4087: Status: Open (was: Patch Available) > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch, 4087.trunk.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242252#comment-13242252 ] Amar Kamat commented on MAPREDUCE-4087: --- Looks good to me. +1 > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242246#comment-13242246 ] Ravi Gummadi commented on MAPREDUCE-4087: - Attached patch is for branch-1. > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4087: Release Note: Fixes the issue of Generate Dist Cache Data generation job slowness. Status: Patch Available (was: Open) > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4087: Attachment: 4087.patch Attaching patch with the fix. > [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases > > > Key: MAPREDUCE-4087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4087.patch > > > In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is > done every time based on the bytes to be written to a distributed cache file. > When we try to write data to next distributed cache file in the same map > task, the size of random data generated in each iteration can become small > based on the particular case. This can make this dist cache data generation > slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4087) [Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases
[Gridmix] GenerateDistCacheData job of Gridmix can become slow in some cases Key: MAPREDUCE-4087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4087 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi In map() method of GenerateDistCacheData job of Gridmix, val.setSize() is done every time based on the bytes to be written to a distributed cache file. When we try to write data to next distributed cache file in the same map task, the size of random data generated in each iteration can become small based on the particular case. This can make this dist cache data generation slow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3879) yarn script has vestiges of jsvc that need to be cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3879: - Resolution: Duplicate Status: Resolved (was: Patch Available) Dup of MAPREDUCE-3578. > yarn script has vestiges of jsvc that need to be cleaned up > --- > > Key: MAPREDUCE-3879 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3879 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-3879.patch.txt > > > If nodemanager is started under root (I know, I know). The following is > displayed: > {noformat} > /usr/lib/jvm/java-6-sun/bin/java -Dproc_nodemanager -Xmx1000m -jvm server > -Dhadoop.log.dir=/var/log/yarn -Dyarn.log.dir=/var/log/yarn > -Dhadoop.log.file=yarn-yarn-nodemanager-ahmed-laptop.log > -Dyarn.log.file=yarn-yarn-nodemanager-ahmed-laptop.log > -Dyarn.home.dir=/usr/lib/hadoop -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop/lib/native > -classpath > /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/share/hadoop/common/*::/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/:/usr/lib/hadoop/share/hadoop/hdfs/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/lib/*:/etc/hadoop/conf/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > Unrecognized option: -jvm > Could not create the Java virtual machine. > {noformat} > The culprit is this bit of code that looks suspiciously like what used to be > in bin/hdfs in support of jsvc launch: > {noformat} > elif [ "$COMMAND" = "nodemanager" ] ; then > CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties > CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager' > if [[ $EUID -eq 0 ]]; then > YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS" > else > YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS" > fi > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3879) yarn script has vestiges of jsvc that need to be cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242221#comment-13242221 ] Devaraj K commented on MAPREDUCE-3879: -- Fixed with MAPREDUCE-3578. > yarn script has vestiges of jsvc that need to be cleaned up > --- > > Key: MAPREDUCE-3879 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3879 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-3879.patch.txt > > > If nodemanager is started under root (I know, I know). The following is > displayed: > {noformat} > /usr/lib/jvm/java-6-sun/bin/java -Dproc_nodemanager -Xmx1000m -jvm server > -Dhadoop.log.dir=/var/log/yarn -Dyarn.log.dir=/var/log/yarn > -Dhadoop.log.file=yarn-yarn-nodemanager-ahmed-laptop.log > -Dyarn.log.file=yarn-yarn-nodemanager-ahmed-laptop.log > -Dyarn.home.dir=/usr/lib/hadoop -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop/lib/native > -classpath > /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/share/hadoop/common/*::/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/:/usr/lib/hadoop/share/hadoop/hdfs/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/lib/*:/etc/hadoop/conf/nm-config/log4j.properties > org.apache.hadoop.yarn.server.nodemanager.NodeManager > Unrecognized option: -jvm > Could not create the Java virtual machine. > {noformat} > The culprit is this bit of code that looks suspiciously like what used to be > in bin/hdfs in support of jsvc launch: > {noformat} > elif [ "$COMMAND" = "nodemanager" ] ; then > CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties > CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager' > if [[ $EUID -eq 0 ]]; then > YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS" > else > YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS" > fi > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira