[jira] [Created] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly
zookeeper artifact is missing from the hadoop-dist assembly --- Key: MAPREDUCE-3996 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Roman Shaposhnik Fix For: 0.23.2 According to maven, zookeeper happens to be a dependency of hadoop-yarn-server-common. Yet it is missing from the final distribution assembly (and hence from the binary tarball) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3980) mr-jobhistory-daemon.sh should look for mapred script in HADOOP_MAPRED_HOME
mr-jobhistory-daemon.sh should look for mapred script in HADOOP_MAPRED_HOME --- Key: MAPREDUCE-3980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3980 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.1 Reporter: Roman Shaposhnik The following: {noformat} nohup nice -n $YARN_NICENESS "$YARN_HOME"/bin/mapred --config $YARN_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null & {noformat} should be this instead: {noformat} nohup nice -n $YARN_NICENESS "$HADOOP_MAPRED_HOME"/bin/mapred --config $YARN_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null & {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3916) various issues with running yarn proxyserver
various issues with running yarn proxyserver Key: MAPREDUCE-3916 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3916 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager, webapps Affects Versions: 0.23.1 Reporter: Roman Shaposhnik Seem like yarn proxyserver is not operational when running out of the 0.23.1 RC2 tarball. # Setting yarn.web-proxy.address to match yarn.resourcemanager.address doesn't disable the proxyserver (althought not setting yarn.web-proxy.address at all correctly disable it and produces a message: org.apache.hadoop.yarn.YarnException: yarn.web-proxy.address is not set so the proxy will not run). This contradicts the documentation provided for yarn.web-proxy.address in yarn-default.xml # Setting yarn.web-proxy.address and running the service results in the following: {noformat} $ ./sbin/yarn-daemon.sh start proxyserver starting proxyserver, logging to /tmp/hadoop-0.23.1/logs/yarn-rvs-proxyserver-ahmed-laptop.out /usr/java/64/jdk1.6.0_22/bin/java -Dproc_proxyserver -Xmx1000m -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir= -Dyarn.id.str=rvs -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/tmp/hadoop-0.23.1/logs -Dyarn.log.dir=/tmp/hadoop-0.23.1/logs -Dhadoop.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.log.file=yarn-rvs-proxyserver-ahmed-laptop.log -Dyarn.home.dir=/tmp/hadoop-0.23.1 -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/tmp/hadoop-0.23.1/lib/native -classpath /tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/etc/hadoop:/tmp/hadoop-0.23.1/share/hadoop/common/lib/*:/tmp/hadoop-0.23.1/share/hadoop/common/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs:/tmp/hadoop-0.23.1/share/hadoop/hdfs/lib/*:/tmp/hadoop-0.23.1/share/hadoop/hdfs/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/*:/tmp/hadoop-0.23.1/share/hadoop/mapreduce/lib/* org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer {noformat} with the following message found in the logs: {noformat} 2012-02-24 09:26:31,099 FATAL org.apache.hadoop.yarn.server.webproxy.WebAppProxy: Could not start proxy web server java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:532) at org.apache.hadoop.http.HttpServer.(HttpServer.java:224) at org.apache.hadoop.http.HttpServer.(HttpServer.java:164) at org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:85) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:76) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3900) mr-jobhistory-daemon.sh should rely on MAPREDUCE env. variables instead of the YARN ones
mr-jobhistory-daemon.sh should rely on MAPREDUCE env. variables instead of the YARN ones Key: MAPREDUCE-3900 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3900 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik It nice to see yarn-deamo.sh be split into a separate script for managing MR service(s), but once that has happened we should go all the way and make it configurable as an MR entity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3879) yarn script has vestiges of jsvc that need to be cleaned up
yarn script has vestiges of jsvc that need to be cleaned up --- Key: MAPREDUCE-3879 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3879 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik If nodemanager is started under root (I know, I know). The following is displayed: {noformat} /usr/lib/jvm/java-6-sun/bin/java -Dproc_nodemanager -Xmx1000m -jvm server -Dhadoop.log.dir=/var/log/yarn -Dyarn.log.dir=/var/log/yarn -Dhadoop.log.file=yarn-yarn-nodemanager-ahmed-laptop.log -Dyarn.log.file=yarn-yarn-nodemanager-ahmed-laptop.log -Dyarn.home.dir=/usr/lib/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/usr/lib/hadoop/lib/native -classpath /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/usr/lib/hadoop/share/hadoop/common/*::/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/:/usr/lib/hadoop/share/hadoop/hdfs/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/*:/usr/lib/hadoop/share/hadoop/mapreduce/lib/*:/etc/hadoop/conf/nm-config/log4j.properties org.apache.hadoop.yarn.server.nodemanager.NodeManager Unrecognized option: -jvm Could not create the Java virtual machine. {noformat} The culprit is this bit of code that looks suspiciously like what used to be in bin/hdfs in support of jsvc launch: {noformat} elif [ "$COMMAND" = "nodemanager" ] ; then CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager' if [[ $EUID -eq 0 ]]; then YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS" else YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS" fi {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3845) hadoop distcp fails to run with java.lang.NoClassDefFoundError
hadoop distcp fails to run with java.lang.NoClassDefFoundError -- Key: MAPREDUCE-3845 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3845 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.1 Reporter: Roman Shaposhnik Here's how to reproduce: {noformat} $ cd /tmnp $ curl http://people.apache.org/~acmurthy/hadoop-0.23.1-rc0/hadoop-0.23.1.tar.gz | tar xzvf - $ cd hadoop-0.23.1 $ ./bin/hadoop distcp Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCp Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.DistCp at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: org.apache.hadoop.tools.DistCp. Program will exit. {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3791) can't build site in hadoop-yarn-server-common
can't build site in hadoop-yarn-server-common - Key: MAPREDUCE-3791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3791 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 0.23.1 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.23.1 Here's how to reproduce: {noformat} $ mvn site site:stage -DskipTests -DskipTest -DskipITs main: [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main SUCCESS [49.017s] [INFO] Apache Hadoop Project POM . SUCCESS [5.152s] [INFO] Apache Hadoop Annotations . SUCCESS [4.973s] [INFO] Apache Hadoop Project Dist POM SUCCESS [4.514s] [INFO] Apache Hadoop Assemblies .. SUCCESS [4.334s] [INFO] Apache Hadoop Auth SUCCESS [5.215s] [INFO] Apache Hadoop Auth Examples ... SUCCESS [4.051s] [INFO] Apache Hadoop Common .. SUCCESS [4.111s] [INFO] Apache Hadoop Common Project .. SUCCESS [4.198s] [INFO] Apache Hadoop HDFS SUCCESS [5.373s] [INFO] Apache Hadoop HttpFS .. SUCCESS [22.549s] [INFO] Apache Hadoop HDFS Project SUCCESS [4.440s] [INFO] hadoop-yarn ... SUCCESS [5.250s] [INFO] hadoop-yarn-api ... SUCCESS [4.579s] [INFO] hadoop-yarn-common SUCCESS [4.268s] [INFO] hadoop-yarn-server SUCCESS [4.408s] [INFO] hadoop-yarn-server-common . FAILURE [0.035s] [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-web-proxy .. SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] Apache Hadoop MapReduce Streaming . SKIPPED [INFO] Apache Hadoop Distributed Copy SKIPPED [INFO] Apache Hadoop Archives SKIPPED [INFO] Apache Hadoop Rumen ... SKIPPED [INFO] Apache Hadoop Extras .. SKIPPED [INFO] Apache Hadoop Tools Dist .. SKIPPED [INFO] Apache Hadoop Tools ... SKIPPED [INFO] Apache Hadoop Distribution SKIPPED [INFO] Apache Hadoop Client .. SKIPPED [INFO] Apache Hadoop Mini-Cluster SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 2:18.427s [INFO] Finished at: Thu Feb 02 10:31:35 PST 2012 [INFO] Final Memory: 321M/1012M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (default) on project hadoop-yarn-server-common: An Ant BuildException has occured: Warning: Could not find file /home/rvs/src/apache/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml to copy. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-yarn-server-common {noformat} -- This message is automatically gener
[jira] [Created] (MAPREDUCE-3728) ShuffleHandler can't access results when configured in a secure mode
ShuffleHandler can't access results when configured in a secure mode Key: MAPREDUCE-3728 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3728 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Fix For: 0.23.1 While running the simplest of jobs (Pi) on MR2 in a fully secure configuration I have noticed that the job was failing on the reduce side with the following messages littering the nodemanager logs: {noformat} 2012-01-19 08:35:32,544 ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find usercache/rvs/appcache/application_1326928483038_0001/output/attempt_1326928483038_0001_m_03_0/file.out.index in any of the configured local directories {noformat} While digging further I found out that the permissions on the files/dirs were prohibiting nodemanager (running under the user yarn) to access these files: {noformat} $ ls -l /data/3/yarn/usercache/testuser/appcache/application_1327102703969_0001/output/attempt_1327102703969_0001_m_01_0 -rw-r- 1 testuser testuser 28 Jan 20 15:41 file.out -rw-r- 1 testuser testuser 32 Jan 20 15:41 file.out.index {noformat} Digging even further revealed that the group-sticky bit that was faithfully put on all the subdirectories between testuser and application_1327102703969_0001 was gone from output and attempt_1327102703969_0001_m_01_0. Looking into how these subdirectories are created (org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs()) {noformat} // $x/usercache/$user/appcache/$appId/filecache Path appFileCacheDir = new Path(appBase, FILECACHE); appsFileCacheDirs[i] = appFileCacheDir.toString(); lfs.mkdir(appFileCacheDir, null, false); // $x/usercache/$user/appcache/$appId/output lfs.mkdir(new Path(appBase, OUTPUTDIR), null, false); {noformat} Reveals that lfs.mkdir ends up manipulating permissions and thus clears sticky bit from output and filecache. At this point I'm at a loss about how this is supposed to work. My understanding was that the whole sequence of events here was predicated on a sticky bit set so that daemons running under the user yarn (default group yarn) can have access to the resulting files and subdirectories down at output and below. Please let me know if I'm missing something or whether this is just a bug that needs to be fixed. On a related note, when the shuffle side of the Pi job failed the job itself didn't. It went into the endless loop and only exited when it exhausted all the local storage for the log files (at which point the nodemanager died and thus the job ended). Perhaps this is even more serious side effect of this issue that needs to be investigated separately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3693) mapreduce's MapReduceChildJVM no longer proxies value of java.library.path
mapreduce's MapReduceChildJVM no longer proxies value of java.library.path --- Key: MAPREDUCE-3693 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Fix For: 0.23.1 I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts). This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that: {noformat} String libraryPath = System.getProperty("java.library.path"); if (libraryPath == null) { libraryPath = workDir.getAbsolutePath(); } else { libraryPath += SYSTEM_PATH_SEPARATOR + workDir; } boolean hasUserLDPath = false; for(int i=0; ihttps://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3635) Improve Hadoop subcomponent integration in Hadoop 0.23
Improve Hadoop subcomponent integration in Hadoop 0.23 -- Key: MAPREDUCE-3635 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3635 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.23.1 Please see HADOOP-7939 for a complete description and discussion. This JIRA is for patch tracking purposes only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding
src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding Key: MAPREDUCE-3374 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.205.0 Reporter: Roman Shaposhnik ant task-controller fails because src/c++/task-controller/configure is not set executable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3319) multifilewc from hadoop examples seems to be broken in 0.20.205.0
multifilewc from hadoop examples seems to be broken in 0.20.205.0 - Key: MAPREDUCE-3319 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3319 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 0.20.205.0 Reporter: Roman Shaposhnik {noformat} /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop/hadoop-examples-0.20.205.0.22.jar multifilewc examples/text examples-output/multifilewc 11/10/31 16:50:26 INFO mapred.FileInputFormat: Total input paths to process : 2 11/10/31 16:50:26 INFO mapred.JobClient: Running job: job_201110311350_0220 11/10/31 16:50:27 INFO mapred.JobClient: map 0% reduce 0% 11/10/31 16:50:42 INFO mapred.JobClient: Task Id : attempt_201110311350_0220_m_00_0, Status : FAILED java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable at org.apache.hadoop.mapred.lib.LongSumReducer.reduce(LongSumReducer.java:44) at org.apache.hadoop.mapred.Task$OldCombinerRunner.combine(Task.java:1431) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1436) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:249) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira