Hey, I am not aware of this error. Maybe ur Hadoop home is not correctly set or some other minor issue.
Sent from my iPhone On 07-Nov-2014, at 2:02 am, "Jing Fan" <fanjin...@gmail.com<mailto:fanjin...@gmail.com>> wrote: Hi Tripti, The GiraphApplicationMaster error but the yarn error log says java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2079) at org.apache.giraph.conf.ClassConfOption.get(ClassConfOption.java:128) at org.apache.giraph.utils.ConfigurationUtils.getTypesHolderClass(ConfigurationUtils.java:178) at org.apache.giraph.conf.GiraphTypes.readFrom(GiraphTypes.java:103) at org.apache.giraph.conf.GiraphClasses.<init>(GiraphClasses.java:161) at org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.<init>(ImmutableClassesGiraphConfiguration.java:138) at org.apache.giraph.yarn.GiraphApplicationMaster.<init>(GiraphApplicationMaster.java:158) at org.apache.giraph.yarn.GiraphApplicationMaster.main(GiraphApplicationMaster.java:431) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2071) ... 7 more Caused by: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045) ... 8 more when running yarn jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/jing/shortestpaths -w 1 -yj giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar There is definitely the class there. Do you know why？ On Thu, Nov 6, 2014 at 8:16 AM, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> wrote: I don't know if u have access to this node. But if u do, u could check if the file is indeed there and u have access to it. Sent from my iPhone On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: You are right it works, but now I receive the following error: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCDynamicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) ... 6 more That justify the other error I receive in the task: Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask Thanks, Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> ha scritto: Why r u adding two jars? Example jar ideally contains core library so everything should be available with just one example jar included Sent from my iPhone On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi, now it seems better, I need to add: giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar Now it seems that after a lot of cycle it fail with this error: Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask But in this case the error appear in task-3-stderr.log not in gam-stderr.log where there is the following error: LF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDynamicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Could not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0006/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) ... 6 more Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> ha scritto: Hi, I remember seeing this thread, but since i was already deep into fixing this issue in production, I didn't really wait for the patch, not even the munge symbol as I was not even able to compile the yarn profile( sorry for being so ruthless to giraph :-( ) I did later cross check with the patch and it was quite similar to my change but sure it wouldn't work for all profiles(in fact some profiles may even break due to Hadoop class incompatibility ) I can share those changes with u once I get back to my computer but I don't think that could have caused this problem. Just try the small changed I mentioned in my previous reply and let me know if that helped. Thanks, Tripti Sent from my iPhone On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi Tripti, let me add one info, As you can see from this thread of email: http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E I had some issue while compiling for Hadoop 2.5.1 could be that into the process I loose something? Thanks again, Alessandro Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> ha scritto: Hi Tripti, yes I run in this way: $HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1 -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar And this is the complete trace: 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from SCDynamicStore 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=10.61.137.28:59677<http://10.61.137.28:59677>, nodeAddress 10.61.137.28:8042<http://10.61.137.28:8042>, nodeRackName /default-rack, nodeNumContainers 0 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415264041937_0005 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id<http://mapred.job.id> is deprecated. Instead, use mapreduce.job.id<http://mapreduce.job.id> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the application master 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name . 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking . 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005_000001, State: ACCEPTED, Containers used: 1 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415264041937_0005 failed 2 times due to AM Container for appattempt_1415264041937_0005_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds. ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_1415264041937_0005 As you can see there is this piece: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar But I continue to receive the error: Thanks, Alessandro Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> ha scritto: For GiraphApplicationMaster not found, the simple thing that worked for me was adding a -yj parameter to my command. (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} {bunch of parameters} -yj {jar-file-name}) If u look at job logs, Do u see a log statement defining "LIB JARS"? Sent from my iPhone On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi, yea my logs shows that error. I follow this example: http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E without success. Then I try to add all jars I can from giraph adding the following lines into the hadoop-env.sh: export MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB when I run the application now I see this: 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph/lib But again the same error. What I think is that there is no problem in the library but while is starting up. I saw this piece in the logs: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs That show that it is able to load GiraphApplicationMaster but fail during the startup. Where can I find more logs? Thanks, Alessandro Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> ha scritto: Can u run the following command to find container logs and post it here: yarn logs -applicationId application_1415203231170_0002 If u find this error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster It may just be due to non-availablity of the yarn jars Thanks, Tripti Tech Yahoo From: Jing Fan <fanjin...@gmail.com<mailto:fanjin...@gmail.com>> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>> Date: Thursday, November 6, 2014 at 1:56 AM To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.apache.org<mailto:user@giraph.apache.org>> Subject: Re: Help with Giraph on Yarn Then do you know to solve this error? 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with exitCode: 1 due to: Exception from container-launch. Container id: container_1415203231170_0002_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi, in the mean time I found the reason why it hangs for long time here: base path checking . It scan all the directory tree starting from current directory, if you run it into your home (and you have a lot of files and directories) it takes ages to complete. I continue to got the same error. Regards, Alessandro Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjin...@gmail.com<mailto:fanjin...@gmail.com>> ha scritto: I run with: hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/jing/shortestpaths -w 1 -yj giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: This is my complete logs with the command I run: ale:~ />$HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths -w 1 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from SCDynamicStore 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032/> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=192.168.0.105:50658<http://192.168.0.105:50658/>, nodeAddress 192.168.0.105:8042<http://192.168.0.105:8042/>, nodeRackName /default-rack, nodeNumContainers 0 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB. 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415189069846_0004 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id<http://mapred.job.id/> is deprecated. Instead, use mapreduce.job.id<http://mapreduce.job.id/> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the application master 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/common/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/common 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/yarn 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application application_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 secs 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415189069846_0004 failed 2 times due to AM Container for appattempt_1415189069846_0004_000002 exited with exitCode: 1 due to: Exception from container-launch: ExitCodeException exitCode=1: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds. Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <tri...@yahoo-inc.com<mailto:tri...@yahoo-inc.com>> ha scritto: I faced the similar problem where the workflow just hangs. Can u see in ur logs if there is a task with id 0. I made a local fix and I could give u the patch if u see a similar problem and ur workflow is unable to find the zookeeper manager file and repeats the following message: For task xyz, got file 'null' Thanks, Tripti Sent from my iPhone On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi, have you seen the hadoop logs? Could you write here how are you running the command? Thanks, Alessandro Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjin...@gmail.com<mailto:fanjin...@gmail.com>> ha scritto: Yeah, I add -yj option. The result is the same. Still error here. On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <alenegr...@yahoo.it<mailto:alenegr...@yahoo.it>> wrote: Hi Jing, I’m facing the same problem. Do you check the hadoop logs? In my case I got: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationMaster Moreover I don’t know why my application hang for a long time here: 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . And then continue failing with your error into the output and the other into the hadoop logs. Did you test adding this? -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar Thanks, Alessandro Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjin...@gmail.com<mailto:fanjin...@gmail.com>> ha scritto: Hi, I install the Giraph with Yarn. But when I run the program with following command: hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/j/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/j/shortestpaths -w 1 The program exited with error: 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 secs 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 secs 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1415203231170_0002 failed 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited with exitCode: 1 due to: Exception from container-launch. Container id: container_1415203231170_0002_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Anyone can help?
