RE: Can't find main class when run ResourceManager or NodeManager
Hi, Please configure HADOOP_COMMON_HOME and HADOOP_HDFS_HOME environment variables. And then start the resource manager it will start. Reason: All the common classes will be loaded to classpath from the HADOOP_COMMON_HOME. So Configuration class is not found in the classpath. Regards, Vinayakumar B. *** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! -Original Message- From: Haoyuan Li [mailto:haoyuan...@gmail.com] Sent: Thursday, August 11, 2011 1:57 AM To: mapreduce-dev@hadoop.apache.org Subject: Can't find main class when run ResourceManager or NodeManager Hi, When I ran ResourceManager or NodeManager as steps here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INST ALL . It has main class can't be found exception... I attached the shell output here. Any help will be appreciated! Thanks, Haoyuan haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan- resourcemanager-hya.out /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../l ogs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../log s -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../l ogs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../log s -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath /home/haoyuan/hadoop/conf:/home/haoyuan/hadoop/conf:/usr/lib/jvm/java-6-sun/ lib/tools.jar:/home/haoyuan/hadoop/trunk/common/hadoop-*.jar:/home/haoyuan/h adoop/trunk/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/ common/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/lib/*.jar :/home/haoyuan/hadoop/trunk/common/share/hadoop/hdfs/*.jar:/home/haoyuan/had oop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/ haoyuan/hadoop/trunk/hdfs/build/classes:/home/haoyuan/hadoop/trunk/hdfs/hado op-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trun k/mapreduce/build/classes:/home/haoyuan/hadoop/trunk/mapreduce/build:/home/h aoyuan/hadoop/trunk/mapreduce/build/test/classes:/home/haoyuan/hadoop/trunk/ mapreduce/build/tools:/home/haoyuan/hadoop/trunk/mapreduce/lib/*.jar:/home/h aoyuan/hadoop/trunk/mapreduce/*.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1. 0-SNAPSHOT/bin/../lib/aopalliance-1.0.jar:/home/haoyuan/hadoop/hadoop-mapred uce-1.0-SNAPSHOT/bin/../lib/asm-3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduc e-1.0-SNAPSHOT/bin/../lib/aspectjrt-1.6.5.jar:/home/haoyuan/hadoop/hadoop-ma preduce-1.0-SNAPSHOT/bin/../lib/avro-1.3.2.jar:/home/haoyuan/hadoop/hadoop-m apreduce-1.0-SNAPSHOT/bin/../lib/avro-1.4.1.jar:/home/haoyuan/hadoop/hadoop- mapreduce-1.0-SNAPSHOT/bin/../lib/clover-3.0.2.jar:/home/haoyuan/hadoop/hado op-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-1.7.0.jar:/home/haoyu an/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-core-1. 8.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/common s-cli-1.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/ commons-codec-1.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin /../lib/commons-collections-3.2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce- 1.0-SNAPSHOT/bin/../lib/commons-configuration-1.6.jar:/home/haoyuan/hadoop/h adoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-digester-1.8.jar:/home/haoyu an/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-httpclient-3.1.ja r:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-lang -2.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commo ns-logging-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/ ../lib/commons-logging-api-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1 .0-SNAPSHOT/bin/../lib/commons-math-2.1.jar:/home/haoyuan/hadoop/hadoop-mapr educe-1.0-SNAPSHOT/bin/../lib/commons-net-1.4.1.jar:/h
MRv2 Configuration Parameters
Hi, I noticed that that yarn-default.xml (mapreduce/yarn/yarn-server/yarn-server-common/src/main/resources/yarn-default.xml) has the configurable parameters for yarn. The installation instructions (1) requires to add the nodemanager.auxiluary.services and nodemanager.aux.service.mapreduce.shuffle.class properties to the yarn-site.xml. What is the purpose of these parameters? Are all the properties for MRv2 with the descriptions and default values mentioned some where in the documentation? It would be nice to update the build process something like what Cloudera has done (2) to figure out the what configuration parameters have been added in a particular release and prepare documentation for the same. Also, the patch owner can be sent a mail to add description for the configuration parameter. Thanks, Praveen (1) - http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL (2) - http://www.cloudera.com/blog/2011/08/automatically-documenting-apache-hadoop-configuration/
Re: Can't find main class when run ResourceManager or NodeManager
Thank you for your reply Praveen. I tried, but got the same error... Best, Haoyuan On Wed, Aug 10, 2011 at 6:01 PM, Praveen Sripati wrote: > Haoyuan, > > Try the following > > cd $HADOOP_COMMON_HOME/lib > ln -s $HADOOP_COMMON_HOME/target/hadoop-common-0.23.0-SNAPSHOT.jar > > For me the HADOOP_COMMON_HOME is set to > > /home/praveensripati/Hadoop/trunk/hadoop-common > > Thanks, > Praveen > > On Thu, Aug 11, 2011 at 1:57 AM, Haoyuan Li wrote: > > > Hi, > > > > When I ran ResourceManager or NodeManager as steps here > > > > > http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL > > . > > It has main class can't be found exception... I attached the shell output > > here. Any help will be appreciated! > > > > Thanks, > > > > Haoyuan > > > > haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh > > start resourcemanager > > starting resourcemanager, logging to > > > > > /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan-resourcemanager-hya.out > > /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m > > > > > -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > > > > -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log > > -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= > > -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA > > -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml > > > > > -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > > > > -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log > > -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log > > -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. > > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath > > > > > /home/haoyuan/hadoop/conf:/home/haoyuan/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/haoyuan/hadoop/trunk/common/hadoop-*.jar:/home/haoyuan/hadoop/trunk/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/hdfs/*.jar:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/hdfs/build/classes:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/build/classes:/home/haoyuan/hadoop/trunk/mapreduce/build:/home/haoyuan/hadoop/trunk/mapreduce/build/test/classes:/home/haoyuan/hadoop/trunk/mapreduce/build/tools:/home/haoyuan/hadoop/trunk/mapreduce/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/*.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aopalliance-1.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/asm-3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aspectjrt-1.6.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/clover-3.0.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-1.7.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-cli-1.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-codec-1.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-collections-3.2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-configuration-1.6.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-digester-1.8.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-httpclient-3.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-lang-2.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-api-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-math-2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-net-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/core-3.1.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guava-r09.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-servlet-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/hadoop-annotations-0.23.0-SNAPSHOT.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jackson-core-asl-1.4.2.jar:/home/haoyuan/hadoo
Re: Can't find main class when run ResourceManager or NodeManager
Haoyuan, Try the following cd $HADOOP_COMMON_HOME/lib ln -s $HADOOP_COMMON_HOME/target/hadoop-common-0.23.0-SNAPSHOT.jar For me the HADOOP_COMMON_HOME is set to /home/praveensripati/Hadoop/trunk/hadoop-common Thanks, Praveen On Thu, Aug 11, 2011 at 1:57 AM, Haoyuan Li wrote: > Hi, > > When I ran ResourceManager or NodeManager as steps here > > http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL > . > It has main class can't be found exception... I attached the shell output > here. Any help will be appreciated! > > Thanks, > > Haoyuan > > haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh > start resourcemanager > starting resourcemanager, logging to > > /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan-resourcemanager-hya.out > /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m > > -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log > -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= > -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA > -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml > > -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > > -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs > -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log > -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log > -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. > -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath > > /home/haoyuan/hadoop/conf:/home/haoyuan/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/haoyuan/hadoop/trunk/common/hadoop-*.jar:/home/haoyuan/hadoop/trunk/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/hdfs/*.jar:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/hdfs/build/classes:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/build/classes:/home/haoyuan/hadoop/trunk/mapreduce/build:/home/haoyuan/hadoop/trunk/mapreduce/build/test/classes:/home/haoyuan/hadoop/trunk/mapreduce/build/tools:/home/haoyuan/hadoop/trunk/mapreduce/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/*.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aopalliance-1.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/asm-3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aspectjrt-1.6.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/clover-3.0.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-1.7.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-cli-1.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-codec-1.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-collections-3.2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-configuration-1.6.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-digester-1.8.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-httpclient-3.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-lang-2.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-api-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-math-2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-net-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/core-3.1.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guava-r09.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-servlet-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/hadoop-annotations-0.23.0-SNAPSHOT.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jackson-core-asl-1.4.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jackson-mapper-asl-1.4.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jdiff-1.0.9.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jets3t-0.6.1.jar:/home/haoyuan/hadoop/hadoop
[jira] [Created] (MAPREDUCE-2804) "Creation of symlink to attempt log dir failed." message is not useful
"Creation of symlink to attempt log dir failed." message is not useful -- Key: MAPREDUCE-2804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2804 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.204.0 Reporter: Allen Wittenauer Priority: Blocker In attempting to qualify the 204 RC2 release, my tasktracker logs are filled with the above message. I'd love to do something about it, but since it doesn't tell me what exactly it is trying to symlink I cannot unless I dig into the source code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2803) Separate client and server configs
Separate client and server configs -- Key: MAPREDUCE-2803 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2803 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Luke Lu Fix For: 0.23.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2689) InvalidStateTransisiton when AM is not assigned to a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2689. -- Resolution: Fixed Fixed as part of recent RM re-factor > InvalidStateTransisiton when AM is not assigned to a job > > > Key: MAPREDUCE-2689 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2689 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil > Fix For: 0.23.0 > > > In cases where an AM is not being assigned to a job, RELEASED at COMPLETED > invalid event is observed. This is easily reproducible in cases such as > MAPREDUCE-2687. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Can't find main class when run ResourceManager or NodeManager
Hi, When I ran ResourceManager or NodeManager as steps here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL . It has main class can't be found exception... I attached the shell output here. Any help will be appreciated! Thanks, Haoyuan haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan-resourcemanager-hya.out /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath /home/haoyuan/hadoop/conf:/home/haoyuan/hadoop/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/haoyuan/hadoop/trunk/common/hadoop-*.jar:/home/haoyuan/hadoop/trunk/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/hdfs/*.jar:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/hdfs/build/classes:/home/haoyuan/hadoop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/build/classes:/home/haoyuan/hadoop/trunk/mapreduce/build:/home/haoyuan/hadoop/trunk/mapreduce/build/test/classes:/home/haoyuan/hadoop/trunk/mapreduce/build/tools:/home/haoyuan/hadoop/trunk/mapreduce/lib/*.jar:/home/haoyuan/hadoop/trunk/mapreduce/*.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aopalliance-1.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/asm-3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/aspectjrt-1.6.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/avro-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/clover-3.0.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-1.7.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-cli-1.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-codec-1.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-collections-3.2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-configuration-1.6.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-digester-1.8.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-httpclient-3.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-lang-2.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-logging-api-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-math-2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-net-1.4.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/core-3.1.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guava-r09.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/guice-servlet-2.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/hadoop-annotations-0.23.0-SNAPSHOT.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jackson-core-asl-1.4.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jackson-mapper-asl-1.4.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jdiff-1.0.9.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jets3t-0.6.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jetty-6.1.26.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jetty-util-6.1.26.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/jsp-api-2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/junit-4.8.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/
[jira] [Created] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[MR-279] Jobhistory filenames should have jobID to help in better parsing -- Key: MAPREDUCE-2802 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 For jobID such as job_1312933838300_0007, jobhistory file names are named as job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2801) Include the native libs in java.library.path
Include the native libs in java.library.path - Key: MAPREDUCE-2801 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2801 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 For the child tasks in mrv2, java.library.path is set to just $PWD and the native libs are not included. Whereas in 0.20.x, java.library.path for child tasks was set to :$PWD -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2800) clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files
clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 in jhist files - Key: MAPREDUCE-2800 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2800 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 clockSplits, cpuUsages, vMemKbytes, physMemKbytes is set to -1 for all the map tasks for the last 4 progress interval in the jobhistory files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2799) [MR-279] NPE is throwing on job -status
[
https://issues.apache.org/jira/browse/MAPREDUCE-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramya Sunil resolved MAPREDUCE-2799.
Resolution: Duplicate
Devaraj, there is already a known bug MAPREDUCE-2686 for the same. Hence
resolving this as duplicate.
> [MR-279] NPE is throwing on job -status
> -
>
> Key: MAPREDUCE-2799
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2799
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>
> {code:xml}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:113)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:101)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:94)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:384)
> at
> org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:223)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2799) [MR-279] NPE is throwing on job -status
[MR-279] NPE is throwing on job -status
-
Key: MAPREDUCE-2799
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2799
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
{code:xml}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.mapred.ClientServiceDelegate.refreshProxy(ClientServiceDelegate.java:113)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:101)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getRefreshedProxy(ClientServiceDelegate.java:94)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:384)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:515)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:154)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:223)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1074)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: MRv2 Configuration in Eclipse
Arun, I did try it out for sometime, but didn't work. Need to get more into the code to fix the problem. It's real difficult to use some editor instead of an IDE to do the development, bug fixing or get familiar with the code. I was able to run the original MapReduce applications from within Eclipse. I created a JIRA for the same. https://issues.apache.org/jira/browse/MAPREDUCE-2798 Thanks, Praveen On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy wrote: > Praveen, > > I'm not aware of any devs using eclipse to run MRv2, I haven't tried it > myself. > > OTOH, if you do get to we are happy to follow your lead, please feel free > to provide us instructions and we can update the README... thanks. > > Arun > > On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote: > > > Hi, > > > > Are there any instructions to start the ResourceManager, NodeManager, > > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran > the > > sample program. The projects have also been exported in Eclipse. I want > to > > run the RM, NM and HS from Eclipse, so as to see the flow and fix any > > problems in the code. > > > > Thanks, > > Praveen > >
[jira] [Created] (MAPREDUCE-2798) Make the ResourceManager, NodeManager and HistoryServer run from Eclipse.
Make the ResourceManager, NodeManager and HistoryServer run from Eclipse. - Key: MAPREDUCE-2798 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2798 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2 Affects Versions: 0.23.0 Reporter: praveen sripati Priority: Minor Fix For: 0.23.0 Make the ResourceManager, NodeManager and HistoryServer run from Eclipse, so that it would be easy for development. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: Subscription for contribution in map-reduce.
Hi Balakrishnan, Please go through this below link to find the process of contributing to Apache Hadoop projects. http://wiki.apache.org/hadoop/HowToContribute Devaraj K -Original Message- From: Balakrishnan Prasanna [mailto:balkiprasa...@gmail.com] Sent: Wednesday, August 10, 2011 3:19 PM To: mapreduce-dev@hadoop.apache.org Subject: Subscription for contribution in map-reduce. Hi, I would like to contribute for map-reduce algorithm. Can I please know what is the procedures ? Regards, Prasanna.
Subscription for contribution in map-reduce.
Hi, I would like to contribute for map-reduce algorithm. Can I please know what is the procedures ? Regards, Prasanna.
[jira] [Created] (MAPREDUCE-2797) Some java files cannot be compiled
Some java files cannot be compiled -- Key: MAPREDUCE-2797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2797 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Tsz Wo (Nicholas), SZE Due to the changes in HDFS-2239, the following files cannot be compiled (Thanks Uma for pointing them out.) 1. src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java 2. src/test/mapred/org/apache/hadoop/mapreduce/security/TestBinaryTokenFile.java 3. src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCacheOldApi.java 4. src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRaid.java -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
