[jira] [Commented] (MAPREDUCE-3151) Contrib tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123391#comment-13123391 ] Joep Rottinghuis commented on MAPREDUCE-3151: - Does the build need to be poked separately? It did not seem to pick up the last commit. > Contrib tests failing > - > > Key: MAPREDUCE-3151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3151 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/vertica >Affects Versions: 0.22.0 > Environment: Jenkins running ant 1.7 (not an issue on ant 1.8) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3151-branch-0.22.patch, > MAPREDUCE-3151-branch-0.22.patch > > > Jenkins builds fail: > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/80/console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123344#comment-13123344 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1062 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1062/]) Reverted MAPREDUCE-3003 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180283 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123335#comment-13123335 ] Hadoop QA commented on MAPREDUCE-3123: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498264/MR-3123.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/974//console This message is automatically generated. > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecuto
[jira] [Updated] (MAPREDUCE-3151) Contrib tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-3151: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 patch looks good. I have just committed it. Thanks Joep. > Contrib tests failing > - > > Key: MAPREDUCE-3151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3151 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/vertica >Affects Versions: 0.22.0 > Environment: Jenkins running ant 1.7 (not an issue on ant 1.8) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3151-branch-0.22.patch, > MAPREDUCE-3151-branch-0.22.patch > > > Jenkins builds fail: > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/80/console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123325#comment-13123325 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Hdfs-trunk-Commit #1121 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1121/]) Reverted MAPREDUCE-3003 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180283 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123324#comment-13123324 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Common-trunk-Commit #1043 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1043/]) Reverted MAPREDUCE-3003 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180283 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reopened MAPREDUCE-3003: --- reverted commit (svn r 1180283) as it breaks MR2 assembly > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3151) Contrib tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123319#comment-13123319 ] Hadoop QA commented on MAPREDUCE-3151: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498263/MAPREDUCE-3151-branch-0.22.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/973//console This message is automatically generated. > Contrib tests failing > - > > Key: MAPREDUCE-3151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3151 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/vertica >Affects Versions: 0.22.0 > Environment: Jenkins running ant 1.7 (not an issue on ant 1.8) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3151-branch-0.22.patch, > MAPREDUCE-3151-branch-0.22.patch > > > Jenkins builds fail: > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/80/console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3123: --- Status: Patch Available (was: Open) Add required apache header and minor test cleanup. > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-09-27 20:58:48,
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3123: --- Attachment: MR-3123.3.patch > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-09-27 20:58:48,951 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: >
[jira] [Updated] (MAPREDUCE-3151) Contrib tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated MAPREDUCE-3151: Attachment: MAPREDUCE-3151-branch-0.22.patch Additional ivy fixes for slf4j dependency for a few contribs. > Contrib tests failing > - > > Key: MAPREDUCE-3151 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3151 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/vertica >Affects Versions: 0.22.0 > Environment: Jenkins running ant 1.7 (not an issue on ant 1.8) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3151-branch-0.22.patch, > MAPREDUCE-3151-branch-0.22.patch > > > Jenkins builds fail: > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/80/console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3123: --- Attachment: MR-3123.2.patch > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch, MR-3123.2.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-09-27 20:58:48,951 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: > 2011-09-27 20:58
[jira] [Created] (MAPREDUCE-3155) Document and likewise implement relevant checks/escape functions for what form of input is acceptable for setting up a container launch context i.e. special chars in
Document and likewise implement relevant checks/escape functions for what form of input is acceptable for setting up a container launch context i.e. special chars in resource names, env vars and the commands --- Key: MAPREDUCE-3155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3155 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Fix For: 0.23.0 What should a user of yarn escape/not escape when passing in input for the container launch contexts - localized resources' names are used to create symlinks. - Are special chars supported in symlinks or do they need to be escaped? - Likewise for environment variables and commands. Current implementation uses a shell script to setup the environment and launch the commands. What should the user be aware of when setting up the launch context? The input also should be such that a user should not need to change code based on what platform or flavor of shell is being used to setup the env and run the commands. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3123: - Status: Open (was: Patch Available) you forgot to add the apache header to the new file. > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-09-27 20:58:48,951 INFO > org.apache.hadoop
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123293#comment-13123293 ] Hitesh Shah commented on MAPREDUCE-3123: Will open a separate jira to track what should be done for escaping user input ( commands, filenames ) in general before we use/execute them. > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3123: --- Status: Patch Available (was: Open) > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011-09-27 20:58:48,951 INFO > org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: > 2011-09-27 20:58:48,951
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3123: --- Attachment: MR-3123.1.patch Attaching a simple patch for now which addresses the immediate bug related to symlinks with special chars. > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2011
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123289#comment-13123289 ] Hadoop QA commented on MAPREDUCE-2876: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498251/MAPREDUCE-2876-branch-0_23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/972//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/972//console This message is automatically generated. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch, > MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123280#comment-13123280 ] Hadoop QA commented on MAPREDUCE-3153: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498250/MAPREDUCE-3153.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/971//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/971//console This message is automatically generated. > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3153.patch > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123269#comment-13123269 ] Mahadev konar commented on MAPREDUCE-2876: -- +1 looks good to me. Will wait for hudson before checkin it in. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch, > MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-2876: --- Status: Patch Available (was: Open) > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch, > MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[
https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anupam Seth updated MAPREDUCE-2876:
---
Attachment: MAPREDUCE-2876-branch-0_23.patch
Thanks Mahadev!
I removed also
{code}
this.monitorInterval = expireIntervel/3;
{code}
from
{code}AbstractLivelinessMonitor#setExpireInterval{code}
and moved it out to the subclasses to avoid the undesirable effect of silently
setting a config param that they might have set apriori in the subclass by
doing something like
{code}
...
setMonitorInterval(expireIntvl/5);
setExpireInterval(expireIntvl);
...
{code}
> ContainerAllocationExpirer appears to use the incorrect configs
> ---
>
> Key: MAPREDUCE-2876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Robert Joseph Evans
>Assignee: Anupam Seth
>Priority: Critical
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2876-branch-0_23.patch,
> MAPREDUCE-2876-branch-0_23.patch
>
>
> ContainerAllocationExpirer sets the expiration interval to be
> RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses
> AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different
> from what AMLivelinessMonitor does.
> There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL
> for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the
> expiry.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3153: - Attachment: MAPREDUCE-3153.patch Fixed the test case to the correct paths in fileoutputcommitter. > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3153.patch > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3153: - Status: Patch Available (was: Open) > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3153.patch > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3153: - Component/s: mrv2 > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123245#comment-13123245 ] Eric Payne commented on MAPREDUCE-2789: --- Thanks Arun. I'll look into the changes. > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.v1.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123223#comment-13123223 ] Hudson commented on MAPREDUCE-2802: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1060 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1060/]) MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123224#comment-13123224
]
Hudson commented on MAPREDUCE-3033:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1060 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1060/])
MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn
frameworks. Contributed by Hitesh Shah.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180214
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2876: - Target Version/s: 0.23.0 (was: 0.24.0, 0.23.0) Affects Version/s: (was: 0.24.0) Status: Open (was: Patch Available) Cancelling patch while Mahadev's comments are addressed. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123216#comment-13123216 ] Arun C Murthy commented on MAPREDUCE-2789: -- Also, # Please don't change JobStatus to include YARN stuff - just return values from old MR framework > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.v1.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Status: Open (was: Patch Available) Eric, sorry it took me this long to review this. Some comments: # Rename JobReport to ApplicationResourceUsageReport or some such - we have tried to avoid 'job' in YARN # Move the JobReport generation to RMAppAttemptImpl and call that from RMAppImpl # I don't get why you have readFields/writeFields in org.apache.hadoop.yarn.api.records.JobResourceInfo interface - we should avoid this. None of the other interface in org.apache.hadoop.yarn.api.records have this. > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.v1.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[
https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123199#comment-13123199
]
Mahadev konar commented on MAPREDUCE-2876:
--
sorry just noticed,
I think we should leave the api still there:
{code}
- protected void setMonitorInterval(int monitorInterval) {
-this.monitorInterval = monitorInterval;
- }
-
{code}
Someone might want to use it with a different monitor interval, other than
default (expiry/3).
> ContainerAllocationExpirer appears to use the incorrect configs
> ---
>
> Key: MAPREDUCE-2876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Anupam Seth
>Priority: Critical
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2876-branch-0_23.patch
>
>
> ContainerAllocationExpirer sets the expiration interval to be
> RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses
> AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different
> from what AMLivelinessMonitor does.
> There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL
> for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the
> expiry.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123193#comment-13123193 ] Mahadev konar commented on MAPREDUCE-2876: -- thanks anupam and bobby, Ill go ahead and commit this. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3087: Attachment: MAPREDUCE-3087.patch Arun. It was a blocker for me to do any dev work because all my jobs were failing I don't know if we ever want to have something in the CLASSPATH which doesn't make sense to the JVM, and /somepath/mr-apps.jar!/mrapp-generated-classpath doesn't make sense to the JVM (so it will be ignored). Please ignore the last patch. It was an attempt to include tests, but it seems pretty hard because: 1. If I include the real mr-app.jar (using pom.xml), then a circular dependency is created. 2. If I don't include the mr-app.jar, then I must checkin and include in the CLASSPATH some jar containing a file mrapp-generated-classpath. This seemed too much effort for a simple one line change. I'm re-uploading the 1st patch with is a single line change. > CLASSPATH not the same after MAPREDUCE-2880 > --- > > Key: MAPREDUCE-3087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, > MAPREDUCE-3087.patch, notworking.classpath.uniq, working.classpath.uniq > > > After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123191#comment-13123191 ] Jonathan Eagles commented on MAPREDUCE-2802: Thanks so much, Arun! > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123188#comment-13123188 ] Hudson commented on MAPREDUCE-2802: --- Integrated in Hadoop-Common-trunk-Commit #1041 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1041/]) MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123186#comment-13123186 ] Hudson commented on MAPREDUCE-2802: --- Integrated in Hadoop-Hdfs-trunk-Commit #1119 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1119/]) MAPREDUCE-2802. Ensure JobHistory filenames have jobId. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180220 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2802: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Jonathan! (Sid - thanks for the review too!) > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123164#comment-13123164 ] Robert Joseph Evans commented on MAPREDUCE-2876: Thanks for the explanation +1 for the patch. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123162#comment-13123162
]
Hudson commented on MAPREDUCE-3033:
---
Integrated in Hadoop-Common-trunk-Commit #1040 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1040/])
MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn
frameworks. Contributed by Hitesh Shah.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180214
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.
[jira] [Commented] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123161#comment-13123161
]
Hudson commented on MAPREDUCE-3033:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1118 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1118/])
MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn
frameworks. Contributed by Hitesh Shah.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180214
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Master.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestMaster.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.Prog
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[
https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123160#comment-13123160
]
Robert Joseph Evans commented on MAPREDUCE-2858:
At Arun's request I am posting the design comments that Luke started on before
he left for IBM.
{quote}
In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and
NameNode (NN)) run as "trusted"
system users, the application masters (AM) run as users who submit the
application. While this offers great flexibility
to run multiple version of mapreduce frameworks (including their UI) on the
same Hadoop cluster, it has significant
implication for the security of webapps (Please do not discuss specific
vulnerabilities here).
Requirements:
0. Secure authentication for AM (for app/job level ACLs).
1. Webapp security should be optional via site configuration.
2. Support existing pluggable single sign on mechanisms.
3. Should not require per app/user configuration for deployment.
4. Should not require special site-wide DNS configuration for deployment.
Proposed design/solution/advisory:
0. The webapp security is turned on by a configuration property:
yarn.server.app-proxy, which is a comma separated list
of host:port addresses. e.g., app-proxy1.cluster1.company.com:8181
1. If any app proxy (AP) is configured, RM will use one of the APs to construct
links for AM webapps. e.g.,
http://app-proxy1.cluster1.company.com:8181/yarn/:app_id/:am_uri.
2. People are advised against visiting a link that points to a random/unknown
host:port in the cluster.
3. AP runs as a system user and acts as an authentication server and pass the
request (:am_uri) and the user.name to
corresponding AMs.
4. AM verifies AP requests by checking whether the request IP address matches
the configured AP list.
5. AP runs as a share-nothing server that gets (and caches) the :app_id to AM
mapping from RM.
6. RM verifies AP requests by checking whether the request IP address matches
the configured AP list.
7. Harden (check referrer etc.) system webapps (run as system user (hadoop,
hdfs or mapred etc.) inside ResourceManager
(RM), NodeManager (NM) and AppProxy (AP) against XSS.
{quote}
{quote}
A brief design note on javascript whitelisting AM proxy (AP):
1. At mapreduce build time, write a program to generate crypto signatures (say
SHA-1) of the content of the script
elements. Since the style of of js usage in MRv2 is fairly standard, this
captures 99.9% of the use cases.
2. At AP start up time, the whitelist of the signatures are loaded in a hashmap.
3. At serving time, the AP fetches pages from AMs and compute the signatures of
the script content (filtering out
anything in the on* attributes as well.) and lookup the hashmap. If all the
script content is whitelisted, AP pass the
page through, otherwise, a UI is displayed with button asking user/admin to
confirm or deny the content.
{quote}
> MRv2 WebApp Security
>
>
> Key: MAPREDUCE-2858
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2, security
>Affects Versions: 0.23.0
>Reporter: Luke Lu
>Assignee: Luke Lu
>Priority: Blocker
> Fix For: 0.23.0
>
>
> In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and
> NameNode (NN)) run as "trusted"
> system users, the application masters (AM) run as users who submit the
> application. While this offers great flexibility
> to run multiple version of mapreduce frameworks (including their UI) on the
> same Hadoop cluster, it has significant
> implication for the security of webapps (Please do not discuss company
> specific vulnerabilities here).
> Requirements:
> # Secure authentication for AM (for app/job level ACLs).
> # Webapp security should be optional via site configuration.
> # Support existing pluggable single sign on mechanisms.
> # Should not require per app/user configuration for deployment.
> # Should not require special site-wide DNS configuration for deployment.
> This the top jira for webapp security. A design doc/notes of threat-modeling
> and counter measures will be posted on the wiki.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3033:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Hitesh!
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123154#comment-13123154 ] Mahadev konar commented on MAPREDUCE-3153: -- Finally found it. Its getting harder to debug the ant stuff. MAPREDUCE-2702 changed the temporary output directories (adding the AM attempt number to the directory structure). The test will have to be fixed to look for the temporary files in that directory. Fix coming. > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3087: - Status: Open (was: Patch Available) Ravi, thanks for digging through this. Patch looks ok, not sure if there is a better way. OTOH, I don't get the changes to pom deps? Also, pls fix the javac warning. Thanks. > CLASSPATH not the same after MAPREDUCE-2880 > --- > > Key: MAPREDUCE-3087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, > notworking.classpath.uniq, working.classpath.uniq > > > After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123122#comment-13123122 ] Mahadev konar commented on MAPREDUCE-3153: -- Its not ivy cache. Its failing for me as well. Will post more details. > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123101#comment-13123101 ] Robert Joseph Evans commented on MAPREDUCE-2876: All I really care about is a unit test to verify that the change is working correctly. If TestNmLivelenessMonitor covers it then that is OK by me. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work logged] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel\#worklog-{worklog.getId()}
]
Prashant Sharma logged work on MAPREDUCE-3131:
--
Author: Prashant Sharma
Created on: 07/Oct/11 19:27
Start Date: 07/Oct/11 19:26
Worklog Time Spent: 48h
Issue Time Tracking
---
Worklog Id: (was: 12135)
Time Spent: 48h
Remaining Estimate: 120h (was: 168h)
> Docs and Scripts for setting up single node MRV2 cluster.
> --
>
> Key: MAPREDUCE-3131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: documentation, mrv2
>Affects Versions: 0.24.0
>Reporter: Prashant Sharma
>Assignee: Prashant Sharma
>Priority: Trivial
> Labels: documentation, hadoop
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch,
> MAPREDUCE-3131.patch, MAPREDUCE-3131.patch
>
> Original Estimate: 168h
> Time Spent: 48h
> Remaining Estimate: 120h
>
> Scripts to run a single node cluster with a default configuration. Takes care
> of running all the daemons including hdfs and yarn.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123097#comment-13123097 ] Anupam Seth commented on MAPREDUCE-2876: There is a test TestNmLivelinessMonitor that seems to do a fairly thorough job of testing expiry. NMLivenessMonitor is a subclass from AbstractLivenessMonitor and AMLivenessMonitor and ContainerAllocatorExpirer do almost the same thing. Please let me know if you would like me to add tests for AMLivenessMonitor and ContainerAllocatorExpirer or enhance coverage of this existing test in some way. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123096#comment-13123096 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1058 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1058/]) MAPREDUCE-3003. Publish Yarn and MapReduce artifacts to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180155 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123081#comment-13123081 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Hdfs-trunk-Commit #1116 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1116/]) MAPREDUCE-3003. Publish Yarn and MapReduce artifacts to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180155 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123076#comment-13123076 ] Hudson commented on MAPREDUCE-3003: --- Integrated in Hadoop-Common-trunk-Commit #1038 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1038/]) MAPREDUCE-3003. Publish Yarn and MapReduce artifacts to Maven snapshot repository. (tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180155 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/pom.xml > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3003: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3003: -- Affects Version/s: 0.24.0 0.23.0 Fix Version/s: 0.24.0 0.23.0 > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123070#comment-13123070 ] Alejandro Abdelnur commented on MAPREDUCE-3003: --- committed to trunk, in a few days it will be committed to 0.23 in a combo with other maven related patches. > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Tom White >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123069#comment-13123069 ] Hadoop QA commented on MAPREDUCE-2988: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498213/MR-2988.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/970//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/970//console This message is automatically generated. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, > MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123054#comment-13123054
]
Hadoop QA commented on MAPREDUCE-3033:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12498206/MR-3033.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/968//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/968//console
This message is automatically generated.
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoo
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123055#comment-13123055 ] Hadoop QA commented on MAPREDUCE-2876: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498198/MAPREDUCE-2876-branch-0_23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/969//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/969//console This message is automatically generated. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Attachment: MR-2988.txt > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, > MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, > MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George reassigned MAPREDUCE-3124:
--
Assignee: John George (was: Thomas Graves)
> mapper failed with failed to load native libs
> -
>
> Key: MAPREDUCE-3124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: John George
>Priority: Blocker
> Fix For: 0.23.0
>
>
> hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
> -job=* -Dmapreduce.map.output.compress=true
> -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
> -Dmapreduce.output.fileoutputformat.compress=true
> -Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
> reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
> -outKey
> org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
> Compression/textinput Compression/textoutput-1317315994
> This will fail with native libs not found error unless
> -Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
> is added.
> The error in container log:
> 2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
> to load the custom-built native-hadoop
> library...2011-09-29 17:06:56,787 DEBUG
> org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
> error: java.lang.UnsatisfiedLinkError: no hadoop in
> java.library.path2011-09-29 17:06:56,787 DEBUG
> org.apache.hadoop.util.NativeCodeLoader:
> java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
> 17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform...
> using builtin-java classes where applicable
> Also note that the error that shows up at the application master for this is
> terrible:
> Container killed by the ApplicationMaster. Container killed on request. Exit
> code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-3033:
---
Status: Patch Available (was: Open)
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-3033:
---
Attachment: MR-3033.2.patch
Addressed code review comments. Also added a minor change in ReduceTask to
default to yarn.
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2876: --- Target Version/s: 0.23.0, 0.24.0 Affects Version/s: 0.24.0 Status: Patch Available (was: Open) Putting into patch available to kick Jenkins to get the test results. The patch mostly looks good. Are there any tests for AbstractLivenessMonitor? If not could you add in a simple one. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123013#comment-13123013
]
Abhijit Suresh Shingate commented on MAPREDUCE-3154:
{quote}
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{quote}
Only *_checkSpecs(job);_* call is moved to beginning of
JobSubmitter.submitJobInternal(Job, Cluster) method.
> Validate the Jobs Output Specification as the first statement in
> JobSubmitter.submitJobInternal(Job, Cluster) method
>
>
> Key: MAPREDUCE-3154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.0, 0.24.0
> Environment: mrv2
>Reporter: Abhijit Suresh Shingate
>Assignee: Abhijit Suresh Shingate
> Attachments: MAPREDUCE-3154.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Presently the output specification is validated after getting new JobId from
> ClientRMService, Copying the job jar, Configuration file, archives etc.
> Instead of that move following Job Output specification validation call to
> the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
> {code}
> checkSpecs(job);
> {code}
> This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3014) Rename and invert logic of '-cbuild' profile to 'native' and off by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123011#comment-13123011 ] Hudson commented on MAPREDUCE-3014: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1056 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1056/]) MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off by default. Contributed by Alejandro Abdelnur. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180116 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/README * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm > Rename and invert logic of '-cbuild' profile to 'native' and off by default > --- > > Key: MAPREDUCE-3014 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3014 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3014.patch > > > This would align MR modules with common & hdfs modules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar reassigned MAPREDUCE-3131: Assignee: Prashant Sharma > Docs and Scripts for setting up single node MRV2 cluster. > -- > > Key: MAPREDUCE-3131 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.24.0 >Reporter: Prashant Sharma >Assignee: Prashant Sharma >Priority: Trivial > Labels: documentation, hadoop > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch, > MAPREDUCE-3131.patch, MAPREDUCE-3131.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Scripts to run a single node cluster with a default configuration. Takes care > of running all the daemons including hdfs and yarn. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122998#comment-13122998
]
Hadoop QA commented on MAPREDUCE-3154:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12498193/MAPREDUCE-3154.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//console
This message is automatically generated.
> Validate the Jobs Output Specification as the first statement in
> JobSubmitter.submitJobInternal(Job, Cluster) method
>
>
> Key: MAPREDUCE-3154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.0, 0.24.0
> Environment: mrv2
>Reporter: Abhijit Suresh Shingate
>Assignee: Abhijit Suresh Shingate
> Attachments: MAPREDUCE-3154.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Presently the output specification is validated after getting new JobId from
> ClientRMService, Copying the job jar, Configuration file, archives etc.
> Instead of that move following Job Output specification validation call to
> the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
> {code}
> checkSpecs(job);
> {code}
> This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth updated MAPREDUCE-2876: --- Attachment: MAPREDUCE-2876-branch-0_23.patch Kindly review patch. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3014) Rename and invert logic of '-cbuild' profile to 'native' and off by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122989#comment-13122989 ] Hudson commented on MAPREDUCE-3014: --- Integrated in Hadoop-Hdfs-trunk-Commit #1115 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1115/]) MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off by default. Contributed by Alejandro Abdelnur. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180116 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/README * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm > Rename and invert logic of '-cbuild' profile to 'native' and off by default > --- > > Key: MAPREDUCE-3014 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3014 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3014.patch > > > This would align MR modules with common & hdfs modules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3014) Rename and invert logic of '-cbuild' profile to 'native' and off by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122988#comment-13122988 ] Hudson commented on MAPREDUCE-3014: --- Integrated in Hadoop-Common-trunk-Commit #1037 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1037/]) MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off by default. Contributed by Alejandro Abdelnur. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180116 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/README * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm > Rename and invert logic of '-cbuild' profile to 'native' and off by default > --- > > Key: MAPREDUCE-3014 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3014 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3014.patch > > > This would align MR modules with common & hdfs modules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) build failed because autoreconf was not run before clean. The make file did not exist yet to run clean from. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3014) Rename and invert logic of '-cbuild' profile to 'native' and off by default
[ https://issues.apache.org/jira/browse/MAPREDUCE-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3014: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk (will commit to 0.23 later with other mvn-related changes). Thanks, Alejandro. > Rename and invert logic of '-cbuild' profile to 'native' and off by default > --- > > Key: MAPREDUCE-3014 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3014 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.23.0, 0.24.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3014.patch > > > This would align MR modules with common & hdfs modules. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhijit Suresh Shingate updated MAPREDUCE-3154:
---
Attachment: MAPREDUCE-3154.patch
> Validate the Jobs Output Specification as the first statement in
> JobSubmitter.submitJobInternal(Job, Cluster) method
>
>
> Key: MAPREDUCE-3154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.0, 0.24.0
> Environment: mrv2
>Reporter: Abhijit Suresh Shingate
>Assignee: Abhijit Suresh Shingate
> Attachments: MAPREDUCE-3154.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Presently the output specification is validated after getting new JobId from
> ClientRMService, Copying the job jar, Configuration file, archives etc.
> Instead of that move following Job Output specification validation call to
> the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
> {code}
> checkSpecs(job);
> {code}
> This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhijit Suresh Shingate updated MAPREDUCE-3154:
---
Status: Patch Available (was: Open)
> Validate the Jobs Output Specification as the first statement in
> JobSubmitter.submitJobInternal(Job, Cluster) method
>
>
> Key: MAPREDUCE-3154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.0, 0.24.0
> Environment: mrv2
>Reporter: Abhijit Suresh Shingate
>Assignee: Abhijit Suresh Shingate
> Attachments: MAPREDUCE-3154.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Presently the output specification is validated after getting new JobId from
> ClientRMService, Copying the job jar, Configuration file, archives etc.
> Instead of that move following Job Output specification validation call to
> the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
> {code}
> checkSpecs(job);
> {code}
> This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
Validate the Jobs Output Specification as the first statement in
JobSubmitter.submitJobInternal(Job, Cluster) method
Key: MAPREDUCE-3154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, mrv2
Affects Versions: 0.23.0, 0.24.0
Environment: mrv2
Reporter: Abhijit Suresh Shingate
Assignee: Abhijit Suresh Shingate
Presently the output specification is validated after getting new JobId from
ClientRMService, Copying the job jar, Configuration file, archives etc.
Instead of that move following Job Output specification validation call to the
begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
{code}
checkSpecs(job);
{code}
This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122952#comment-13122952 ] Alejandro Abdelnur commented on MAPREDUCE-3003: --- planning to commit this JIRA in a couple of hours. > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3003) Publish Yarn and MapReduce artifacts to Maven snapshot repository
[ https://issues.apache.org/jira/browse/MAPREDUCE-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned MAPREDUCE-3003: - Assignee: Alejandro Abdelnur (was: Tom White) > Publish Yarn and MapReduce artifacts to Maven snapshot repository > - > > Key: MAPREDUCE-3003 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3003 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Reporter: Tom White >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3003-0.23.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, MAPREDUCE-3003.patch, > MAPREDUCE-3003.patch > > > Currently this is failing since no distribution management section is defined > in the POM. > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/883/consoleFull -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122918#comment-13122918 ] Hadoop QA commented on MAPREDUCE-2988: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498178/MR-2988.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/966//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/966//console This message is automatically generated. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122900#comment-13122900 ] Jonathan Eagles commented on MAPREDUCE-2802: Thanks, Sid! > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122897#comment-13122897 ] Hudson commented on MAPREDUCE-2751: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1054 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1054/]) MAPREDUCE-2751. Modified NodeManager to stop leaving around local files after application finishes. Contributed by Siddharth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180071 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Attachment: MR-2988.txt OK I have addressed all of the issues/comments so far. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122876#comment-13122876 ] Hudson commented on MAPREDUCE-2751: --- Integrated in Hadoop-Common-trunk-Commit #1035 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1035/]) MAPREDUCE-2751. Modified NodeManager to stop leaving around local files after application finishes. Contributed by Siddharth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180071 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2696) Container logs aren't getting cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2696: --- Affects Version/s: 0.23.0 Fix Version/s: 0.23.0 > Container logs aren't getting cleaned up > > > Key: MAPREDUCE-2696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2696 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > > Container logs aren't getting cleaned up causing NMs to fill up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2696) Container logs aren't getting cleaned up
[ https://issues.apache.org/jira/browse/MAPREDUCE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned MAPREDUCE-2696: -- Assignee: Siddharth Seth (was: Vinod Kumar Vavilapalli) Assigning it to Sid, who is taking care of most of the pending stuff in logs' aggregation. > Container logs aren't getting cleaned up > > > Key: MAPREDUCE-2696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2696 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > > Container logs aren't getting cleaned up causing NMs to fill up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122880#comment-13122880 ] Hudson commented on MAPREDUCE-2751: --- Integrated in Hadoop-Hdfs-trunk-Commit #1113 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1113/]) MAPREDUCE-2751. Modified NodeManager to stop leaving around local files after application finishes. Contributed by Siddharth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180071 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2751: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Perfect. I just committed this to trunk and branch-0.23. Thanks Sid! > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3084) race when KILL_CONTAINER is received for a LOCALIZED container
[
https://issues.apache.org/jira/browse/MAPREDUCE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122857#comment-13122857
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3084:
I discussed with Sid regarding this and found that there is bigger race here.
ContainerLaunch does a bunch of setup steps needed for the container and if a
kill arrives during any of it, the container will never be killed. We do cancel
the Future, but that is without any interrupts. Also we try to kill the
process, but the PID may not be available at that point of time.
This part of the code needs a minor redesign.
> race when KILL_CONTAINER is received for a LOCALIZED container
> --
>
> Key: MAPREDUCE-3084
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3084
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Hitesh Shah
>Priority: Blocker
> Attachments: MR-3084.1.patch, MR-3084.wip.patch
>
>
> Depending on when ContainersLaunch starts a container, {{KILL_CONTAINER}}
> when container state is {{LOCALIZED}} ({{LAUNCH_CONTAINER}} event already
> sent) can end up generating a {{CONTAINER_LAUNCHED}} event - which isn't
> handled by ContainerState: {{KILLING}}. Also, the launched container won't be
> killed since {{CLEANUP_CONTAINER}} would have already been processed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122855#comment-13122855 ] Hadoop QA commented on MAPREDUCE-2751: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498165/MR2751_v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/965//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/965//console This message is automatically generated. > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2751: -- Status: Patch Available (was: Open) > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2751) [MR-279] Lot of local files left on NM after the app finish.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2751: -- Attachment: MR2751_v2.patch Update patch with review feedback. > [MR-279] Lot of local files left on NM after the app finish. > > > Key: MAPREDUCE-2751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2751 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR2751.patch, MR2751_v2.patch > > > This ticket is about app-only files which should be cleaned after app-finish. > I see these undeleted after app-finish: > /tmp/nm-local-dir/0/nmPrivate/application_1305091029545_0001/* > /tmp/nm-local-dir/0/nmPrivate/container_1305019205843_0001_02/* > /tmp/nm-local-dir/0/usercache/nobody/appcache/application_1305091029545_0001/* > We should check for other left-over files too, if any. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3144: -- Status: Open (was: Patch Available) Canceling the patch for now, for additional AppMaster information. (MR2923 + containerId, node, etc) > Augment JobHistory to include information needed for serving aggregated logs. > - > > Key: MAPREDUCE-3144 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0 > > Attachments: MR3144.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar reassigned MAPREDUCE-3153: Assignee: Mahadev konar > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anupam Seth reassigned MAPREDUCE-2876: -- Assignee: Anupam Seth (was: Robert Joseph Evans) > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122821#comment-13122821 ] Siddharth Seth commented on MAPREDUCE-2802: --- lgtm. NB +1. Changing the delimiter may be an issue for some apps - but the .23 jobhistory filename has anyway changed, and there are APIs available to pull information from the filename. > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2783) mr279 job history handling after killing application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122820#comment-13122820 ] Eric Payne commented on MAPREDUCE-2783: --- Great! Thanks Vinod! > mr279 job history handling after killing application > > > Key: MAPREDUCE-2783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2783 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Eric Payne >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2783.v1.txt > > Time Spent: 24h > > The job history/application tracking url handling during kill is not > consistent. Currently if you kill a job that was running the tracking url > points to job history, but job history server doesn't have the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2923) Add a new record to JobHistory to record start time of each instance of AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122800#comment-13122800 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2923: Devaraj (K), um, as part of MAPREDUCE-3144, we need more information per AM that needs to be added. So, we are thinking of adding a separate event for an AM. That would rework most of the stuff being done here. Do you mind if I close this as duplicate of that ticket? Saves time and effort, so let me know. Thanks. > Add a new record to JobHistory to record start time of each instance of AM > -- > > Key: MAPREDUCE-2923 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2923 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2923.patch > > > Add a new record to JobHistory to record start time of each instance of AM -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122797#comment-13122797 ] Hadoop QA commented on MAPREDUCE-3144: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498151/MR3144.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/964//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/964//console This message is automatically generated. > Augment JobHistory to include information needed for serving aggregated logs. > - > > Key: MAPREDUCE-3144 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 0.23.0 > > Attachments: MR3144.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122789#comment-13122789 ] Robert Joseph Evans commented on MAPREDUCE-2988: Yes I will clean up the phases of the make. Previously everything was done as part of package. It ran autoreconf, clean, configure and finally compile. ERRORFILE and LOGFILE should actually be added in as part of the next issue MAPREDUCE-2747, but I put it in here for debugging an issue I was seeing (nobody is user id 99 and I had to set the min.user.id to that in my config). In the code that executes this, if the return code is not 0 then everything from stderr is included in the message of the exception that is thrown. Everything from stdout is just logged, and there was a bug in the stdout logging where nothing was being logged, which I fixed. I started putting it into stderr, so I could at least see what was happening, and I forgot to clean it up. And yes I did run into the infinite loop working on the next one, I will backport that fix here. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
