Hi Tripti, The GiraphApplicationMaster error but the yarn error log says
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2079) at org.apache.giraph.conf.ClassConfOption.get(ClassConfOption.java:128) at org.apache.giraph.utils.ConfigurationUtils.getTypesHolderClass(ConfigurationUtils.java:178) at org.apache.giraph.conf.GiraphTypes.readFrom(GiraphTypes.java:103) at org.apache.giraph.conf.GiraphClasses.<init>(GiraphClasses.java:161) at org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.<init>(ImmutableClassesGiraphConfiguration.java:138) at org.apache.giraph.yarn.GiraphApplicationMaster.<init>(GiraphApplicationMaster.java:158) at org.apache.giraph.yarn.GiraphApplicationMaster.main(GiraphApplicationMaster.java:431) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2071) ... 7 more Caused by: java.lang.ClassNotFoundException: Class org.apache.giraph.examples.SimplePageRankComputation not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045) ... 8 more when running yarn jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/jing/shortestpaths -w 1 -yj giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar There is definitely the class there. Do you know why？ On Thu, Nov 6, 2014 at 8:16 AM, Tripti Singh <tri...@yahoo-inc.com> wrote: > I don't know if u have access to this node. But if u do, u could check > if the file is indeed there and u have access to it. > > Sent from my iPhone > > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegr...@yahoo.it> > wrote: > > You are right it works, but now I receive the following error: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from > SCDynamicStore > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: > Could not configure the containerlaunch context for GiraphYarnTasks. > at > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) > at > org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) > at > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) > at > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.FileNotFoundException: File does not exist: > hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0009/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) > at > org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) > at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) > at > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) > ... 6 more > > > That justify the other error I receive in the task: > > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask > > > Thanks, > > Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <tri...@yahoo-inc.com> > ha scritto: > > Why r u adding two jars? Example jar ideally contains core library so > everything should be available with just one example jar included > > Sent from my iPhone > > On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegr...@yahoo.it> > wrote: > > Hi, > now it seems better, I need to add: > > > giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > > Now it seems that after a lot of cycle it fail with this error: > > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask > > But in this case the error appear in task-3-stderr.log not > in gam-stderr.log where there is the following error: > > LF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from > SCDynamicStore > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: > Could not configure the containerlaunch context for GiraphYarnTasks. > at > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:391) > at > org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicationMaster.java:78) > at > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMaster.java:522) > at > org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.FileNotFoundException: File does not exist: > hdfs://hadoop-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_0006/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) > at > org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153) > at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) > at > org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(GiraphApplicationMaster.java:387) > ... 6 more > > > Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tri...@yahoo-inc.com> > ha scritto: > > Hi, > I remember seeing this thread, but since i was already deep into fixing > this issue in production, I didn't really wait for the patch, not even the > munge symbol as I was not even able to compile the yarn profile( sorry for > being so ruthless to giraph :-( ) > I did later cross check with the patch and it was quite similar to my > change but sure it wouldn't work for all profiles(in fact some profiles may > even break due to Hadoop class incompatibility ) I can share those changes > with u once I get back to my computer but I don't think that could have > caused this problem. > Just try the small changed I mentioned in my previous reply and let me > know if that helped. > > Thanks, > Tripti > Sent from my iPhone > > On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegr...@yahoo.it> > wrote: > > Hi Tripti, > let me add one info, As you can see from this thread of email: > > > http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E > > I had some issue while compiling for Hadoop 2.5.1 could be that into the > process I loose something? > > Thanks again, > Alessandro > > > > Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro < > alenegr...@yahoo.it> ha scritto: > > Hi Tripti, > yes I run in this way: > > > $HADOOP_INSTALL/bin/hadoop jar > /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > -vip /tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths > -w 1 -yj > /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > And this is the complete trace: > > 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from > SCDynamicStore > 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://hadoop-master:9000/shortestpaths > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client > 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at / > 0.0.0.0:8032 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM > for, nodeId=10.61.137.28:59677, nodeAddress 10.61.137.28:8042, > nodeRackName /default-rack, nodeNumContainers 0 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap > size to 1024MB. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1415264041937_0005 > 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM > :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources > :: > hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS > :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name . > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking . > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/myLib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/myLib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/lib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /contrib/capacity-scheduler/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /contrib/capacity-scheduler > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application > application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after > submission :application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 > secs > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: > appattempt_1415264041937_0005_000001, State: ACCEPTED, Containers used: 1 > 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415264041937_0005 failed > 2 times due to AM Container for appattempt_1415264041937_0005_000002 exited > with exitCode: 1 due to: Exception from container-launch: > ExitCodeException exitCode=1: > ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > Container exited with a non-zero exit code 1 > .Failing this attempt.. Failing the application. > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed > cache directory for Giraph job. > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total > running time: 0 minutes, 3 seconds. > ale:~/extprj/giraph/myGiraph />yarn logs -applicationId > application_1415264041937_0005 > > As you can see there is this piece: > > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS > :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > > But I continue to receive the error: > > Thanks, > Alessandro > > > Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tri...@yahoo-inc.com> > ha scritto: > > For GiraphApplicationMaster not found, the simple thing that worked for > me was adding a -yj parameter to my command. > (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} > {bunch of parameters} -yj {jar-file-name}) > If u look at job logs, > Do u see a log statement defining "LIB JARS"? > > Sent from my iPhone > > On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegr...@yahoo.it> > wrote: > > Hi, > yea my logs shows that error. I follow this example: > > > http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E > > without success. > Then I try to add all jars I can from giraph adding the following lines > into the hadoop-env.sh: > > export > MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB > > > when I run the application now I see this: > > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/myLib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/myLib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/lib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph/lib > > > But again the same error. What I think is that there is no problem in > the library but while is starting up. > I saw this piece in the logs: > > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: > ApplicationSumbissionContext for GiraphApplicationMaster launch container > is populated. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application > application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after > submission :application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 > secs > > > That show that it is able to load GiraphApplicationMaster but fail > during the startup. > Where can I find more logs? > > Thanks, > Alessandro > > > Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tri...@yahoo-inc.com> > ha scritto: > > Can u run the following command to find container logs and post it here: > > yarn logs -applicationId application_1415203231170_0002 > > If u find this error: > > Could not find or load main class > org.apache.giraph.yarn.GiraphApplicationMaster > > It may just be due to non-availablity of the yarn jars > > Thanks, > Tripti > Tech Yahoo > > From: Jing Fan <fanjin...@gmail.com> > Reply-To: "user@giraph.apache.org" <user@giraph.apache.org> > Date: Thursday, November 6, 2014 at 1:56 AM > To: "user@giraph.apache.org" <user@giraph.apache.org> > Subject: Re: Help with Giraph on Yarn > > Then do you know to solve this error? > > 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415203231170_0002 failed > 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited > with exitCode: 1 due to: Exception from container-launch. > Container id: container_1415203231170_0002_02_000001 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegr...@yahoo.it> > wrote: > >> Hi, >> in the mean time I found the reason why it hangs for long time here: >> >> base path checking . >> >> >> It scan all the directory tree starting from current directory, if you >> run it into your home (and you have a lot of files and directories) it >> takes ages to complete. >> >> I continue to got the same error. >> >> Regards, >> Alessandro >> >> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjin...@gmail.com> >> ha scritto: >> >> I run with: >> >> hadoop jar >> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> -vip /user/jing/tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /user/jing/shortestpaths -w 1 -yj >> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >> >> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegr...@yahoo.it> >> wrote: >> >>> This is my complete logs with the command I run: >>> >>> >>> ale:~ />$HADOOP_INSTALL/bin/hadoop jar >>> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>> -vip /tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /shortestpaths >>> -w 1 >>> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from >>> SCDynamicStore >>> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load >>> native-hadoop library for your platform... using builtin-java classes where >>> applicable >>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format >>> specified. Ensure your InputFormat does not require one. >>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format >>> specified. Ensure your OutputFormat does not require one. >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: >>> hdfs://hadoop-master:9000/shortestpaths >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client >>> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at / >>> 0.0.0.0:8032 >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM >>> for, nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, >>> nodeRackName /default-rack, nodeNumContainers 0 >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap >>> size to 1024MB. >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application >>> ID: application_1415189069846_0004 >>> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is >>> deprecated. Instead, use mapreduce.job.id >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for >>> the application master >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM >>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources >>> :: >>> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml >>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : >>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . >>> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/common >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>> /contrib/capacity-scheduler/*.jar >>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>> /contrib/capacity-scheduler >>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: >>> ApplicationSumbissionContext for GiraphApplicationMaster launch container >>> is populated. >>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to >>> ASM >>> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application >>> application_1415189069846_0004 >>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after >>> submission :application_1415189069846_0004 >>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >>> container request was submitted to ResourceManager for job: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation >>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 >>> secs >>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: >>> appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1 >>> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>> state, diagnostics show: Application application_1415189069846_0004 failed >>> 2 times due to AM Container for appattempt_1415189069846_0004_000002 exited >>> with exitCode: 1 due to: Exception from container-launch: >>> ExitCodeException exitCode=1: >>> ExitCodeException exitCode=1: >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>> at >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) >>> at >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> >>> >>> Container exited with a non-zero exit code 1 >>> .Failing this attempt.. Failing the application. >>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS >>> distributed cache directory for Giraph job. >>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total >>> running time: 0 minutes, 3 seconds. >>> >>> >>> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh < >>> tri...@yahoo-inc.com> ha scritto: >>> >>> I faced the similar problem where the workflow just hangs. >>> Can u see in ur logs if there is a task with id 0. >>> I made a local fix and I could give u the patch if u see a similar >>> problem and ur workflow is unable to find the zookeeper manager file and >>> repeats the following message: >>> For task xyz, got file 'null' >>> >>> Thanks, >>> Tripti >>> Sent from my iPhone >>> >>> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegr...@yahoo.it> >>> wrote: >>> >>> Hi, >>> have you seen the hadoop logs? >>> Could you write here how are you running the command? >>> >>> Thanks, >>> Alessandro >>> >>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjin...@gmail.com> >>> ha scritto: >>> >>> Yeah, I add -yj option. The result is the same. Still error here. >>> >>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <alenegr...@yahoo.it> >>> wrote: >>> >>>> Hi Jing, >>>> I’m facing the same problem. Do you check the hadoop logs? In my case I >>>> got: >>>> >>>> Could not find or load main class >>>> org.apache.giraph.yarn.GiraphApplicationMaster >>>> >>>> Moreover I don’t know why my application hang for a long time here: >>>> >>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>>> >>>> And then continue failing with your error into the output and the >>>> other into the hadoop logs. >>>> >>>> Did you test adding this? >>>> >>>> -yj >>>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>> >>>> Thanks, >>>> Alessandro >>>> >>>> >>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjin...@gmail.com> >>>> ha scritto: >>>> >>>> Hi, >>>> >>>> I install the Giraph with Yarn. >>>> But when I run the program with following command: >>>> >>>> hadoop jar >>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>> org.apache.giraph.GiraphRunner >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>> -vip /user/j/tiny_graph.txt -vof >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>> /user/j/shortestpaths -w 1 >>>> >>>> The program exited with error: >>>> >>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 >>>> secs >>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 >>>> secs >>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 >>>> secs >>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 >>>> secs >>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>>> state, diagnostics show: Application application_1415203231170_0002 failed >>>> 2 times due to AM Container for appattempt_1415203231170_0002_000002 exited >>>> with exitCode: 1 due to: Exception from container-launch. >>>> Container id: container_1415203231170_0002_02_000001 >>>> Exit code: 1 >>>> Stack trace: ExitCodeException exitCode=1: >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>> at >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> Anyone can help? >>>> >>>> >>>> >>> >>> >>> >> >> > > > > > >
