RE: Can't find main class when run ResourceManager or NodeManager
Hi, Please configure HADOOP_COMMON_HOME and HADOOP_HDFS_HOME environment variables. And then start the resource manager it will start. Reason: All the common classes will be loaded to classpath from the HADOOP_COMMON_HOME. So Configuration class is not found in the classpath. Regards, Vinayakumar B. *** This e-mail and attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient's) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! -Original Message- From: Haoyuan Li [mailto:haoyuan...@gmail.com] Sent: Thursday, August 11, 2011 1:57 AM To: mapreduce-dev@hadoop.apache.org Subject: Can't find main class when run ResourceManager or NodeManager Hi, When I ran ResourceManager or NodeManager as steps here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INST ALL . It has main class can't be found exception... I attached the shell output here. Any help will be appreciated! Thanks, Haoyuan haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan- resourcemanager-hya.out /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../l ogs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../log s -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../l ogs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../log s -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath /home/haoyuan/hadoop/conf:/home/haoyuan/hadoop/conf:/usr/lib/jvm/java-6-sun/ lib/tools.jar:/home/haoyuan/hadoop/trunk/common/hadoop-*.jar:/home/haoyuan/h adoop/trunk/common/lib/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/ common/*.jar:/home/haoyuan/hadoop/trunk/common/share/hadoop/common/lib/*.jar :/home/haoyuan/hadoop/trunk/common/share/hadoop/hdfs/*.jar:/home/haoyuan/had oop/trunk/hdfs/hadoop-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/ haoyuan/hadoop/trunk/hdfs/build/classes:/home/haoyuan/hadoop/trunk/hdfs/hado op-*.jar:/home/haoyuan/hadoop/trunk/hdfs/lib/*.jar:/home/haoyuan/hadoop/trun k/mapreduce/build/classes:/home/haoyuan/hadoop/trunk/mapreduce/build:/home/h aoyuan/hadoop/trunk/mapreduce/build/test/classes:/home/haoyuan/hadoop/trunk/ mapreduce/build/tools:/home/haoyuan/hadoop/trunk/mapreduce/lib/*.jar:/home/h aoyuan/hadoop/trunk/mapreduce/*.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1. 0-SNAPSHOT/bin/../lib/aopalliance-1.0.jar:/home/haoyuan/hadoop/hadoop-mapred uce-1.0-SNAPSHOT/bin/../lib/asm-3.2.jar:/home/haoyuan/hadoop/hadoop-mapreduc e-1.0-SNAPSHOT/bin/../lib/aspectjrt-1.6.5.jar:/home/haoyuan/hadoop/hadoop-ma preduce-1.0-SNAPSHOT/bin/../lib/avro-1.3.2.jar:/home/haoyuan/hadoop/hadoop-m apreduce-1.0-SNAPSHOT/bin/../lib/avro-1.4.1.jar:/home/haoyuan/hadoop/hadoop- mapreduce-1.0-SNAPSHOT/bin/../lib/clover-3.0.2.jar:/home/haoyuan/hadoop/hado op-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-1.7.0.jar:/home/haoyu an/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-beanutils-core-1. 8.0.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/common s-cli-1.2.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/ commons-codec-1.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin /../lib/commons-collections-3.2.1.jar:/home/haoyuan/hadoop/hadoop-mapreduce- 1.0-SNAPSHOT/bin/../lib/commons-configuration-1.6.jar:/home/haoyuan/hadoop/h adoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-digester-1.8.jar:/home/haoyu an/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-httpclient-3.1.ja r:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commons-lang -2.5.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../lib/commo ns-logging-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/ ../lib/commons-logging-api-1.0.4.jar:/home/haoyuan/hadoop/hadoop-mapreduce-1 .0-SNAPSHOT/bin/../lib/commons-math-2.1.jar:/home/haoyuan/hadoop/hadoop-mapr
[jira] [Created] (MAPREDUCE-2805) Update RAID for HDFS-2241
Update RAID for HDFS-2241
-
Key: MAPREDUCE-2805
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2805
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Tsz Wo (Nicholas), SZE
{noformat}
src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:44:
interface expected here
[javac] public class RaidBlockSender implements java.io.Closeable,
FSConstants {
[javac]^
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Pre-commit Jenkins job disabled
https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/ Is there a reason the M/R pre-commit build is disabled, while Common and HDFS are enabled? Is this a vestige of the build slaves being offline? -- Aaron T. Myers Software Engineer, Cloudera
[jira] [Created] (MAPREDUCE-2807) MR-279: AM restart does not work after RM refactor
MR-279: AM restart does not work after RM refactor -- Key: MAPREDUCE-2807 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2807 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Sharad Agarwal When the AM crashes, RM is not able to launch a new App attempt. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Can't find main class when run ResourceManager or NodeManager
Report YARN_* and HADOOP_* variables that you've set before starting the daemons. Also run 'bin/yarn classpath' and post the output of that command too. If the output doesn't contain yarn-server-resourcemanager-1.0-SNAPSHOT.jar, you are missing something. +Vinod On Thursday 11 August 2011 01:57 AM, Haoyuan Li wrote: Hi, When I ran ResourceManager or NodeManager as steps here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL . It has main class can't be found exception... I attached the shell output here. Any help will be appreciated! Thanks, Haoyuan haoyuan@hya:~/hadoop/hadoop-mapreduce-1.0-SNAPSHOT$ ./bin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs/yarn-haoyuan-resourcemanager-hya.out /usr/lib/jvm/java-6-sun/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir= -Dyarn.id.str=haoyuan -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -Dyarn.policy.file=hadoop-policy.xml -Dhadoop.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dyarn.log.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/../logs -Dhadoop.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.log.file=yarn-haoyuan-resourcemanager-hya.log -Dyarn.home.dir=/home/haoyuan/hadoop/hadoop-mapreduce-1.0-SNAPSHOT/bin/.. -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA -classpath
[jira] [Resolved] (MAPREDUCE-2787) MR-279: Performance improvement in running Uber MapTasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan resolved MAPREDUCE-2787. - Resolution: Won't Fix Thanks Arun and Vinod for the clarification. I am closing the ticket. MR-279: Performance improvement in running Uber MapTasks Key: MAPREDUCE-2787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-2787.patch The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Commits to MR-279 branch
Hey all, I am in the process of figuring out all the stuff that is need for successfully merging MR-279 to trunk. I am doing this off-line in parallel so that we are ready when other minor issues are being fixed. Right now I am practicing in a private branch that is updated with trunk code and using modified/updated script that Arun uploaded on MR-279 JIRA issue. I now have a version which compiles both yarn+MR (mvn clean install) and classic run-time (ant jar jar-test binary), yay! This is a bit of a big merge, and in particular any changes to hadoop-mapreduce-client-core module in the MR-279 branch will affect my private merge branch. So I request you to be judicious with your commits to MR-279 branch for 4-5 days. In fact, I'd be pleased if you can flag to me any commits that change hadoop-mapreduce-client-core in the MR-279 branch so that I can update my private merge branch correspondingly. Thanks for the help, +Vinod
[jira] [Created] (MAPREDUCE-2808) pull MAPREDUCE-2797 into mr279 branch
pull MAPREDUCE-2797 into mr279 branch - Key: MAPREDUCE-2808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2808 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Fix For: 0.23.0 The ant tar command fails in the mapreduce directory on the mr279 branch. The issue was a change in hdfs and was fixed on trunk with jira MAPREDUCE-2797. Pull that change into mr279. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2821) [MR-279] Missing fields in job summary logs
[MR-279] Missing fields in job summary logs Key: MAPREDUCE-2821 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2821 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.0 The following fields are missing in the job summary logs in mrv2: - numSlotsPerMap - numSlotsPerReduce - clusterCapacity (Earlier known as clusterMapCapacity and clusterReduceCapacity in 0.20.x) The first two fields are important to know if the job was a High RAM job or not and the last field is important to know the total available resource in the cluster during job execution. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2834) [MR-279] Enable dense update for file sink metrics
[MR-279] Enable dense update for file sink metrics -- Key: MAPREDUCE-2834 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2834 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Fix For: 0.23.0 Currently, if File sink is enabled for MRAppMaster or Resourcemanager, it does not populate the file with all the available attributes. It would be useful for debugging and admin purpose to have all the metrics populated in the file. For eg: MRAppMaster metrics currently logs value only for JobsRunning even though the total available job level metrics are JobsCompleted, JobsFailed, JobsKilled, JobsPreparing etc -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2835) Make per-job counter limits configurable
Make per-job counter limits configurable Key: MAPREDUCE-2835 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2835 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.20.204.0 Reporter: Tom White Assignee: Tom White Fix For: 0.20.205.0 The per-job counter limits introduced in MAPREDUCE-1943 are fixed, except for the total number allowed per job (mapreduce.job.counters.limit). It would be useful to make them all configurable. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Can't find main class when run ResourceManager or NodeManager
I get a similar error on Mac OS X. I've built YARN and extracted the tarball to a directory, but when I run bin/yarn-daemon.sh start nodemanager, it prints the following to its log (apologies for the long trace): log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread main org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.yarn.server.nodemanager.NodeManager at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:80) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:146) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:191) Caused by: org.apache.avro.AvroRuntimeException: org.apache.hadoop.yarn.YarnException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:140) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) ... 2 more Caused by: org.apache.hadoop.yarn.YarnException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:70) at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getProxy(HadoopYarnProtoRPC.java:35) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getRMClient(NodeStatusUpdaterImpl.java:158) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:136) ... 3 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:67) ... 7 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/ipc/ClientCache at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine.clinit(ProtoOverHadoopRpcEngine.java:63) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1108) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1160) at org.apache.hadoop.ipc.RPC.getProtocolEngine(RPC.java:94) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:422) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:368) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:333) at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:461) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:442) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.init(ResourceTrackerPBClientImpl.java:32) ... 12 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.ipc.ClientCache at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 24 more In addition, if I try to start resourcemanager instead, it logs: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/matei/workspace/MR-279/common/build/ivy/lib/Hadoop-Common/common/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/matei/workspace/MR-279-deploy/hadoop-mapreduce-1.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at
[jira] [Created] (MAPREDUCE-2836) Provide option to fail jobs when submitted to non-existent pools.
Provide option to fail jobs when submitted to non-existent pools. - Key: MAPREDUCE-2836 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2836 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: Jeff Bean In some environments, it might be desirable to explicitly specify the fair scheduler pools and to explicitly fail jobs that are not submitted to any of the pools. Current behavior of the fair scheduler is to submit jobs to a default pool if a pool name isn't specified or to create a pool with the new name if the pool name doesn't already exist. There should be a configuration option for the fair scheduler that causes it to noisily fail the job if it's submitted to a pool that isn't pre-specified or if the specified pool doesn't exist. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Can't find main class when run ResourceManager or NodeManager
I downloaded the latest version: r1156719. and then redid from the scratch. Now, the nodemanager and run well. However, I got into the same state as Matei when I tried to run ResourceManager. Any help will be appreciated. Best, Haoyuan The followings are output. The output from .out log file = Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:495) = === the output from .log log file === 2011-08-11 15:25:38,357 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Resource Manager is starting... 2011-08-11 15:25:38,878 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher 2011-08-11 15:25:38,879 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher 2011-08-11 15:25:38,880 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher 2011-08-11 15:25:38,881 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher 2011-08-11 15:25:38,904 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType for class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited. 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer is inited. 2011-08-11 15:25:38,935 INFO org.apache.hadoop.yarn.service.AbstractService: Service:AMLivelinessMonitor is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.NodesListManager is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:NMLivelinessMonitor is inited. 2011-08-11 15:25:38,940 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ClientRMService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.AdminService is inited. 2011-08-11 15:25:38,965 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher is inited. 2011-08-11 15:25:38,965 INFO org.apache.hadoop.yarn.service.AbstractService: Service:ResourceManager is inited. 2011-08-11 15:25:39,057 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-08-11 15:25:39,125 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2011-08-11 15:25:39,128 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
Re: Can't find main class when run ResourceManager or NodeManager
Haoyuan, Thats an issue with having avro 1.3.2 jar in the classpath. Please see if you have that version of 1.3.2 jar in the classpath and remove it. We use 1.4 avro jar which isnt compatible with 1.3.2. hope that helps! mahadev On Thu, Aug 11, 2011 at 3:36 PM, Haoyuan Li haoyuan...@gmail.com wrote: I downloaded the latest version: r1156719. and then redid from the scratch. Now, the nodemanager and run well. However, I got into the same state as Matei when I tried to run ResourceManager. Any help will be appreciated. Best, Haoyuan The followings are output. The output from .out log file = Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:495) = === the output from .log log file === 2011-08-11 15:25:38,357 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Resource Manager is starting... 2011-08-11 15:25:38,878 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher 2011-08-11 15:25:38,879 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher 2011-08-11 15:25:38,880 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher 2011-08-11 15:25:38,881 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher 2011-08-11 15:25:38,904 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType for class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited. 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer is inited. 2011-08-11 15:25:38,935 INFO org.apache.hadoop.yarn.service.AbstractService: Service:AMLivelinessMonitor is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.NodesListManager is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:NMLivelinessMonitor is inited. 2011-08-11 15:25:38,940 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ClientRMService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.AdminService is inited. 2011-08-11 15:25:38,965 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher is inited. 2011-08-11 15:25:38,965 INFO org.apache.hadoop.yarn.service.AbstractService: Service:ResourceManager is inited. 2011-08-11 15:25:39,057 INFO org.mortbay.log: Logging to
Re: Can't find main class when run ResourceManager or NodeManager
It looks like the Avro 1.3.2 JAR was included in the tar ball built in the mapreduce project. It's present in lib in the unzipped tarball. However, even if I remove it, the error still happens. Matei On Aug 11, 2011, at 3:47 PM, Mahadev Konar wrote: Haoyuan, Thats an issue with having avro 1.3.2 jar in the classpath. Please see if you have that version of 1.3.2 jar in the classpath and remove it. We use 1.4 avro jar which isnt compatible with 1.3.2. hope that helps! mahadev On Thu, Aug 11, 2011 at 3:36 PM, Haoyuan Li haoyuan...@gmail.com wrote: I downloaded the latest version: r1156719. and then redid from the scratch. Now, the nodemanager and run well. However, I got into the same state as Matei when I tried to run ResourceManager. Any help will be appreciated. Best, Haoyuan The followings are output. The output from .out log file = Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:495) = === the output from .log log file === 2011-08-11 15:25:38,357 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Resource Manager is starting... 2011-08-11 15:25:38,878 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher 2011-08-11 15:25:38,879 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher 2011-08-11 15:25:38,880 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher 2011-08-11 15:25:38,881 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$NodeEventDispatcher 2011-08-11 15:25:38,904 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType for class org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited. 2011-08-11 15:25:38,907 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer is inited. 2011-08-11 15:25:38,935 INFO org.apache.hadoop.yarn.service.AbstractService: Service:AMLivelinessMonitor is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.NodesListManager is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher is inited. 2011-08-11 15:25:38,936 INFO org.apache.hadoop.yarn.service.AbstractService: Service:NMLivelinessMonitor is inited. 2011-08-11 15:25:38,940 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.ClientRMService is inited. 2011-08-11 15:25:38,941 INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.resourcemanager.AdminService is inited. 2011-08-11 15:25:38,965 INFO org.apache.hadoop.yarn.service.AbstractService:
Hadoop-Mapreduce-trunk-Commit - Build # 761 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/761/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2852 lines...] ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy/ivysettings.xml ivy-resolve-common: ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy/ivysettings.xml ivy-resolve-mapred: ivy-retrieve-mapred: init: [touch] Creating /tmp/null262167822 [delete] Deleting: /tmp/null262167822 [unzip] Expanding: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build/ivy/lib/Hadoop/common/hadoop-hdfs-0.23.0-SNAPSHOT.jar into /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build check-c++-configure: create-c++-pipes-configure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1945: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1881: Execute failed: java.io.IOException: Cannot run program autoreconf (in directory /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes): java.io.IOException: error=2, No such file or directory Total time: 3 minutes 3 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/*.jar': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory mv: cannot stat `build/docs/api': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Updating MAPREDUCE-2489 Updating HDFS-2239 Updating MAPREDUCE-2797 Updating MAPREDUCE-2805 Updating HDFS-2241 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Hadoop-Mapreduce-22-branch - Build # 63 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/63/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2275 lines...] A src/examples/org/apache/hadoop/examples/pi/Util.java A src/examples/org/apache/hadoop/examples/pi/package.html A src/examples/org/apache/hadoop/examples/AggregateWordCount.java A src/examples/org/apache/hadoop/examples/Grep.java A bin A bin/mapred-config.sh AUbin/stop-mapred.sh AUbin/mapred AUbin/start-mapred.sh A build-utils.xml A build.xml U. Fetching 'https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/common/src/test/bin' at -1 into '/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/src/test/bin' Checking out http://svn.apache.org/repos/asf/hadoop/nightly AUsrc/test/bin/test-patch.sh A src/test/bin/test-patch.properties At revision 1156857 At revision 1156857 AUtar-munge A commitBuild.sh A hudsonEnv.sh A jenkinsSetup A jenkinsSetup/installTools.sh AUhudsonBuildHadoopNightly.sh A buildMR-279Branch.sh AUhudsonBuildHadoopPatch.sh AUhudsonBuildHadoopRelease.sh AUprocessHadoopPatchEmailRemote.sh AUhudsonPatchQueueAdmin.sh AUprocessHadoopPatchEmail.sh A README.txt A test-patch A test-patch/test-patch.sh At revision 1156857 no change for http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/mapreduce since the previous build no change for https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/common/src/test/bin since the previous build No emails were triggered. [Hadoop-Mapreduce-22-branch] $ /bin/bash /tmp/hudson2277159968527450946.sh + ulimit -n 1024 + export ANT_OPTS=-Xmx2048m + pwd + TRUNK=/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk + cd /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-22-branch/trunk + /homes/hudson/tools/ant/latest/bin/ant -Dversion=2011-08-11_23-35-48 -Declipse.home=/homes/hudson/tools/eclipse/latest -Dfindbugs.home=/homes/hudson/tools/findbugs/latest -Dforrest.home=/homes/hudson/tools/forrest/latest -Dcompile.c++=true -Dcompile.native=true clean create-c++-configure tar findbugs nightly/hudsonBuildHadoopNightly.sh: 1: /homes/hudson/tools/ant/latest/bin/ant: not found + RESULT=127 + [ 127 != 0 ] + echo Build Failed: remaining tests not run Build Failed: remaining tests not run + exit 127 [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Hadoop-Mapreduce-trunk - Build # 752 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/752/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2380 lines...] A bin A bin/mapred-config.sh AUbin/stop-mapred.sh AUbin/mapred AUbin/start-mapred.sh A build-utils.xml A build.xml U. Fetching 'https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin' at -1 into '/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/test/bin' Checking out http://svn.apache.org/repos/asf/hadoop/nightly AUsrc/test/bin/smart-apply-patch.sh AUsrc/test/bin/test-patch.sh At revision 1156857 At revision 1156857 AUtar-munge A commitBuild.sh A hudsonEnv.sh A jenkinsSetup A jenkinsSetup/installTools.sh AUhudsonBuildHadoopNightly.sh A buildMR-279Branch.sh AUhudsonBuildHadoopPatch.sh AUhudsonBuildHadoopRelease.sh AUprocessHadoopPatchEmailRemote.sh AUhudsonPatchQueueAdmin.sh AUprocessHadoopPatchEmail.sh A README.txt A test-patch A test-patch/test-patch.sh At revision 1156857 no change for https://svn.apache.org/repos/asf/hadoop/common/trunk/common/src/test/bin since the previous build No emails were triggered. [Hadoop-Mapreduce-trunk] $ /bin/bash /tmp/hudson3776079390733309452.sh + ulimit -n 1024 + export ANT_OPTS=-Xmx2048m + pwd + TRUNK=/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk + cd /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk + /homes/hudson/tools/ant/latest/bin/ant -Dversion=2011-08-11_23-36-03 -Declipse.home=/homes/hudson/tools/eclipse/latest -Dfindbugs.home=/homes/hudson/tools/findbugs/latest -Dforrest.home=/homes/hudson/tools/forrest/latest -Dcompile.c++=true -Dcompile.native=true clean create-c++-configure tar findbugs nightly/hudsonBuildHadoopNightly.sh: 1: /homes/hudson/tools/ant/latest/bin/ant: not found + RESULT=127 + [ 127 != 0 ] + echo Build Failed: remaining tests not run Build Failed: remaining tests not run + exit 127 [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Updating MAPREDUCE-2489 Updating HDFS-2239 Updating MAPREDUCE-2797 Updating MAPREDUCE-2805 Updating HDFS-2241 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Can't find main class when run ResourceManager or NodeManager
Hi Mahadev, This works for me. Thank you. However, another issue came to me when I tried to run an example. Here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL , it says: $HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars $HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output However, there is no /bin folder in $HADOOP_COMMON_HOME. I found /bin/hadoop in $HADOOP_COMMON_HOME/src/main/bin/hadoop. When I executed the command: ./hadoop-common/src/main/bin/hadoop jar /home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=haoyuan - Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars /home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output I got another exception: === haoyuan@hya:~/hadoop/trunk$ ./hadoop-common/src/main/bin/hadoop jar /home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=haoyuan - Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars /home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/util/RunJar Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.RunJar at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: org.apache.hadoop.util.RunJar. Program will exit. === Any clue? Thank you. Best, Haoyuan On Thu, Aug 11, 2011 at 3:47 PM, Mahadev Konar maha...@hortonworks.comwrote: Haoyuan, Thats an issue with having avro 1.3.2 jar in the classpath. Please see if you have that version of 1.3.2 jar in the classpath and remove it. We use 1.4 avro jar which isnt compatible with 1.3.2. hope that helps! mahadev On Thu, Aug 11, 2011 at 3:36 PM, Haoyuan Li haoyuan...@gmail.com wrote: I downloaded the latest version: r1156719. and then redid from the scratch. Now, the nodemanager and run well. However, I got into the same state as Matei when I tried to run ResourceManager. Any help will be appreciated. Best, Haoyuan The followings are output. The output from .out log file = Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:495) = === the output from .log log file === 2011-08-11 15:25:38,357 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Resource Manager is starting... 2011-08-11 15:25:38,878 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher 2011-08-11 15:25:38,879 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher 2011-08-11 15:25:38,880 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
[jira] [Created] (MAPREDUCE-2837) MR-279: Bug fixes ported from y-merge
MR-279: Bug fixes ported from y-merge - Key: MAPREDUCE-2837 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2837 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arun C Murthy Attachments: rest.patch Similar to MAPREDUCE-2679. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2838) to fix mapreduce builds to use the new hadoop common test jars
to fix mapreduce builds to use the new hadoop common test jars -- Key: MAPREDUCE-2838 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2838 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan maprecude builds are still resolving the old hadoop-common-test jars.. Instead ivy classifiers should be used to resolve the new hadoop-common test jars ; as maven publishes test jars with classifier tests and not as a separate artifact. [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (1979ms) [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-test/0.23.0-SNAPSHOT/hadoop-common-test-0.23.0-20110727.191243-218.jar ... [ivy:resolve] (885kB) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Pre-commit Jenkins job disabled
mr-2838 should be fixed before enabling pre-commit for mr. https://issues.apache.org/jira/browse/MAPREDUCE-2838 -Giri On Thu, Aug 11, 2011 at 12:41 AM, Aaron T. Myers a...@cloudera.com wrote: https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/ Is there a reason the M/R pre-commit build is disabled, while Common and HDFS are enabled? Is this a vestige of the build slaves being offline? -- Aaron T. Myers Software Engineer, Cloudera
Re: Can't find main class when run ResourceManager or NodeManager
Hoayuan,
RunJar is in the hadoop-common-0.23.0-SNAPSHOT.jar. Do a 'hadoop classpath'
and check if the jar file is there in the classpath location.
Similarly, running 'yarn classpath' will provide the classpath for running
the yarn daemons (RM, NM and HS).
Thanks,
Praveen
On Fri, Aug 12, 2011 at 5:18 AM, Haoyuan Li haoyuan...@gmail.com wrote:
Hi Mahadev,
This works for me. Thank you.
However, another issue came to me when I tried to run an example.
Here
http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
,
it says:
$HADOOP_COMMON_HOME/bin/hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.clientfactory.class.name
=org.apache.hadoop.mapred.YarnClientFactory
-Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
However, there is no /bin folder in $HADOOP_COMMON_HOME. I found
/bin/hadoop
in $HADOOP_COMMON_HOME/src/main/bin/hadoop. When I executed the command:
./hadoop-common/src/main/bin/hadoop jar
/home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=haoyuan -
Dmapreduce.clientfactory.class.name
=org.apache.hadoop.mapred.YarnClientFactory
-Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
/home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
I got another exception:
===
haoyuan@hya:~/hadoop/trunk$ ./hadoop-common/src/main/bin/hadoop jar
/home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=haoyuan -
Dmapreduce.clientfactory.class.name
=org.apache.hadoop.mapred.YarnClientFactory
-Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
/home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/util/RunJar
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.RunJar
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.util.RunJar. Program will
exit.
===
Any clue?
Thank you.
Best,
Haoyuan
On Thu, Aug 11, 2011 at 3:47 PM, Mahadev Konar maha...@hortonworks.com
wrote:
Haoyuan,
Thats an issue with having avro 1.3.2 jar in the classpath. Please
see if you have that version of 1.3.2 jar in the classpath and remove
it. We use 1.4 avro jar which isnt compatible with 1.3.2.
hope that helps!
mahadev
On Thu, Aug 11, 2011 at 3:36 PM, Haoyuan Li haoyuan...@gmail.com
wrote:
I downloaded the latest version: r1156719. and then redid from the
scratch.
Now, the nodemanager and run well. However, I got into the same state
as
Matei when I tried to run ResourceManager. Any help will be
appreciated.
Best,
Haoyuan
The followings are output.
The output from .out log file =
Exception in thread main java.lang.IllegalStateException: For this
operation, current State must be STARTED instead of INITED
at
org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101)
at
org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher.stop(ApplicationMasterLauncher.java:90)
at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:423)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:495)
=
=== the output from .log log file ===
2011-08-11 15:25:38,357 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Resource
Manager is starting...
2011-08-11 15:25:38,878 INFO
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType
for class
Hadoop-Mapreduce-trunk-Commit - Build # 762 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/762/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2653 lines...] [get] Not modified - so not downloaded ivy-init-dirs: ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy/ivysettings.xml ivy-resolve-common: ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/ivy/ivysettings.xml ivy-resolve-mapred: ivy-retrieve-mapred: init: [touch] Creating /tmp/null1469506434 [delete] Deleting: /tmp/null1469506434 [unzip] Expanding: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build/ivy/lib/Hadoop/common/hadoop-hdfs-0.23.0-SNAPSHOT.jar into /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build check-c++-configure: create-c++-pipes-configure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1945: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1881: Execute failed: java.io.IOException: Cannot run program autoreconf (in directory /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes): java.io.IOException: error=2, No such file or directory Total time: 23 seconds == == STORE: saving artifacts == == mv: cannot stat `build/*.tar.gz': No such file or directory mv: cannot stat `build/*.jar': No such file or directory mv: cannot stat `build/test/findbugs': No such file or directory mv: cannot stat `build/docs/api': No such file or directory Build Failed [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Publishing Clover coverage report... No Clover report will be published due to a Build Failure Recording test results Publishing Javadoc Recording fingerprints Updating MAPREDUCE-2187 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-2839) MR Jobs fail on a secure cluster with viewfs
MR Jobs fail on a secure cluster with viewfs Key: MAPREDUCE-2839 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2839 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 0.23.0 TokenCache needs to use the new FileSystem.getDelegationTokens api for it to work with viewfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Can't find main class when run ResourceManager or NodeManager
Preveen, Thank you for your reply. The CLASSPATH generated by the '/trunk/hadoop-common/src/main/bin/hadoop' and 'hadoop-config.sh' in my machine is a mess. From my understanding, 'hadoop-config.sh' still generates the CLASSPATH based on the build structure from ant? While now we are using mvn. Could you please tell me the location of your 'hadoop' and 'hadoop-config.sh' scripts? Or maybe you could send me a copy of your hadoop-config.sh, which I suppose should be the same as mine. Not sure whether these could be the reason or not... Any help will be appreciated! Best, Haoyuan On Thu, Aug 11, 2011 at 7:32 PM, Praveen Sripati praveensrip...@gmail.comwrote: Hoayuan, RunJar is in the hadoop-common-0.23.0-SNAPSHOT.jar. Do a 'hadoop classpath' and check if the jar file is there in the classpath location. Similarly, running 'yarn classpath' will provide the classpath for running the yarn daemons (RM, NM and HS). Thanks, Praveen On Fri, Aug 12, 2011 at 5:18 AM, Haoyuan Li haoyuan...@gmail.com wrote: Hi Mahadev, This works for me. Thank you. However, another issue came to me when I tried to run an example. Here http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL , it says: $HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name =org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars $HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output However, there is no /bin folder in $HADOOP_COMMON_HOME. I found /bin/hadoop in $HADOOP_COMMON_HOME/src/main/bin/hadoop. When I executed the command: ./hadoop-common/src/main/bin/hadoop jar /home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=haoyuan - Dmapreduce.clientfactory.class.name =org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars /home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output I got another exception: === haoyuan@hya:~/hadoop/trunk$ ./hadoop-common/src/main/bin/hadoop jar /home/haoyuan/hadoop/trunk/mapreduce/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=haoyuan - Dmapreduce.clientfactory.class.name =org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=1 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars /home/haoyuan/hadoop/trunk/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/util/RunJar Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.RunJar at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Could not find the main class: org.apache.hadoop.util.RunJar. Program will exit. === Any clue? Thank you. Best, Haoyuan On Thu, Aug 11, 2011 at 3:47 PM, Mahadev Konar maha...@hortonworks.com wrote: Haoyuan, Thats an issue with having avro 1.3.2 jar in the classpath. Please see if you have that version of 1.3.2 jar in the classpath and remove it. We use 1.4 avro jar which isnt compatible with 1.3.2. hope that helps! mahadev On Thu, Aug 11, 2011 at 3:36 PM, Haoyuan Li haoyuan...@gmail.com wrote: I downloaded the latest version: r1156719. and then redid from the scratch. Now, the nodemanager and run well. However, I got into the same state as Matei when I tried to run ResourceManager. Any help will be appreciated. Best, Haoyuan The followings are output. The output from .out log file = Exception in thread main java.lang.IllegalStateException: For this operation, current State must be STARTED instead of INITED at org.apache.hadoop.yarn.service.AbstractService.ensureCurrentState(AbstractService.java:101) at org.apache.hadoop.yarn.service.AbstractService.stop(AbstractService.java:69) at
