Hi Tripti, let me add one info, As you can see from this thread of email: http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E
I had some issue while compiling for Hadoop 2.5.1 could be that into the process I loose something? Thanks again, Alessandro Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegr...@yahoo.it> ha scritto: > Hi Tripti, > yes I run in this way: > > > $HADOOP_INSTALL/bin/hadoop jar > /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat > -op /shortestpaths -w 1 -yj > /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > And this is the complete trace: > > 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from > SCDynamicStore > 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://hadoop-master:9000/shortestpaths > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client > 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for, > nodeId=10.61.137.28:59677, nodeAddress 10.61.137.28:8042, nodeRackName > /default-rack, nodeNumContainers 0 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size > to 1024MB. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1415264041937_0005 > 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM > :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS > :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name . > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking . > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/myLib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/myLib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /Users/ale/extprj/giraph/myGiraph/lib/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /Users/ale/extprj/giraph/myGiraph/lib > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name > /contrib/capacity-scheduler/*.jar > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking > /contrib/capacity-scheduler > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application > application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission > :application_1415264041937_0005 > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: > appattempt_1415264041937_0005_000001, State: ACCEPTED, Containers used: 1 > 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415264041937_0005 failed 2 > times due to AM Container for appattempt_1415264041937_0005_000002 exited > with exitCode: 1 due to: Exception from container-launch: ExitCodeException > exitCode=1: > ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > Container exited with a non-zero exit code 1 > .Failing this attempt.. Failing the application. > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed > cache directory for Giraph job. > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total > running time: 0 minutes, 3 seconds. > ale:~/extprj/giraph/myGiraph />yarn logs -applicationId > application_1415264041937_0005 > > As you can see there is this piece: > > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS > :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > > > But I continue to receive the error: > > Thanks, > Alessandro > > > Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tri...@yahoo-inc.com> ha > scritto: > >> For GiraphApplicationMaster not found, the simple thing that worked for me >> was adding a -yj parameter to my command. >> (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} >> {bunch of parameters} -yj {jar-file-name}) >> If u look at job logs, >> Do u see a log statement defining "LIB JARS"? >> >> Sent from my iPhone >> >> On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegr...@yahoo.it> wrote: >> >>> Hi, >>> yea my logs shows that error. I follow this example: >>> >>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E >>> >>> without success. >>> Then I try to add all jars I can from giraph adding the following lines >>> into the hadoop-env.sh: >>> >>> export >>> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar >>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB >>> >>> >>> when I run the application now I see this: >>> >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >>> /opt/yarn/hadoop-2.5.1/share/myLib/*.jar >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking >>> /opt/yarn/hadoop-2.5.1/share/myLib >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >>> /Users/ale/extprj/giraph/myGiraph/*.jar >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking >>> /Users/ale/extprj/giraph/myGiraph >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >>> /Users/ale/extprj/giraph/myGiraph/lib/*.jar >>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking >>> /Users/ale/extprj/giraph/myGiraph/lib >>> >>> >>> But again the same error. What I think is that there is no problem in the >>> library but while is starting up. >>> I saw this piece in the logs: >>> >>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext >>> for GiraphApplicationMaster launch container is populated. >>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM >>> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application >>> application_1415264041937_0005 >>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after >>> submission :application_1415264041937_0005 >>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >>> container request was submitted to ResourceManager for job: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation >>> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: >>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 >>> secs >>> >>> >>> That show that it is able to load GiraphApplicationMaster but fail during >>> the startup. >>> Where can I find more logs? >>> >>> Thanks, >>> Alessandro >>> >>> >>> Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tri...@yahoo-inc.com> >>> ha scritto: >>> >>>> Can u run the following command to find container logs and post it here: >>>> >>>> yarn logs -applicationId application_1415203231170_0002 >>>> >>>> If u find this error: >>>> Could not find or load main class >>>> org.apache.giraph.yarn.GiraphApplicationMaster >>>> It may just be due to non-availablity of the yarn jars >>>> Thanks, >>>> Tripti >>>> Tech Yahoo >>>> >>>> From: Jing Fan <fanjin...@gmail.com> >>>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org> >>>> Date: Thursday, November 6, 2014 at 1:56 AM >>>> To: "user@giraph.apache.org" <user@giraph.apache.org> >>>> Subject: Re: Help with Giraph on Yarn >>>> >>>> Then do you know to solve this error? >>>> >>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>>> state, diagnostics show: Application application_1415203231170_0002 failed >>>> 2 times due to AM Container for appattempt_1415203231170_0002_000002 >>>> exited with exitCode: 1 due to: Exception from container-launch. >>>> Container id: container_1415203231170_0002_02_000001 >>>> Exit code: 1 >>>> Stack trace: ExitCodeException exitCode=1: >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>> at >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>>> at >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegr...@yahoo.it> >>>> wrote: >>>>> Hi, >>>>> in the mean time I found the reason why it hangs for long time here: >>>>> >>>>> base path checking . >>>>> >>>>> >>>>> It scan all the directory tree starting from current directory, if you >>>>> run it into your home (and you have a lot of files and directories) it >>>>> takes ages to complete. >>>>> >>>>> I continue to got the same error. >>>>> >>>>> Regards, >>>>> Alessandro >>>>> >>>>> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjin...@gmail.com> ha >>>>> scritto: >>>>> >>>>>> I run with: >>>>>> >>>>>> hadoop jar >>>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>>>> org.apache.giraph.GiraphRunner >>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>> -vip /user/jing/tiny_graph.txt -vof >>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>>>> /user/jing/shortestpaths -w 1 -yj >>>>>> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>>>> >>>>>> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegr...@yahoo.it> >>>>>> wrote: >>>>>>> This is my complete logs with the command I run: >>>>>>> >>>>>>> >>>>>>> ale:~ />$HADOOP_INSTALL/bin/hadoop jar >>>>>>> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>>>>> org.apache.giraph.GiraphRunner >>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>> -vip /tiny_graph.txt -vof >>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>>>>> /shortestpaths -w 1 >>>>>>> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from >>>>>>> SCDynamicStore >>>>>>> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load >>>>>>> native-hadoop library for your platform... using builtin-java classes >>>>>>> where applicable >>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format >>>>>>> specified. Ensure your InputFormat does not require one. >>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format >>>>>>> specified. Ensure your OutputFormat does not require one. >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: >>>>>>> hdfs://hadoop-master:9000/shortestpaths >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client >>>>>>> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at >>>>>>> /0.0.0.0:8032 >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM >>>>>>> for, nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, >>>>>>> nodeRackName /default-rack, nodeNumContainers 0 >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap >>>>>>> size to 1024MB. >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application >>>>>>> ID: application_1415189069846_0004 >>>>>>> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is >>>>>>> deprecated. Instead, use mapreduce.job.id >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for >>>>>>> the application master >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM >>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... >>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in >>>>>>> LocalResources :: >>>>>>> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml >>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : >>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . >>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name >>>>>>> /contrib/capacity-scheduler/*.jar >>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking >>>>>>> /contrib/capacity-scheduler >>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: >>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch >>>>>>> container is populated. >>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to >>>>>>> ASM >>>>>>> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application >>>>>>> application_1415189069846_0004 >>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after >>>>>>> submission :application_1415189069846_0004 >>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster >>>>>>> container request was submitted to ResourceManager for job: Giraph: >>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation >>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: >>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: >>>>>>> 0,82 secs >>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: >>>>>>> appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: >>>>>>> 1 >>>>>>> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: >>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports >>>>>>> FAILED state, diagnostics show: Application >>>>>>> application_1415189069846_0004 failed 2 times due to AM Container for >>>>>>> appattempt_1415189069846_0004_000002 exited with exitCode: 1 due to: >>>>>>> Exception from container-launch: ExitCodeException exitCode=1: >>>>>>> ExitCodeException exitCode=1: >>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>>>>> at >>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>>>>> at >>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >>>>>>> at >>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) >>>>>>> at >>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>>> >>>>>>> >>>>>>> Container exited with a non-zero exit code 1 >>>>>>> .Failing this attempt.. Failing the application. >>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS >>>>>>> distributed cache directory for Giraph job. >>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: >>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, >>>>>>> total running time: 0 minutes, 3 seconds. >>>>>>> >>>>>>> >>>>>>> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh >>>>>>> <tri...@yahoo-inc.com> ha scritto: >>>>>>> >>>>>>>> I faced the similar problem where the workflow just hangs. >>>>>>>> Can u see in ur logs if there is a task with id 0. >>>>>>>> I made a local fix and I could give u the patch if u see a similar >>>>>>>> problem and ur workflow is unable to find the zookeeper manager file >>>>>>>> and repeats the following message: >>>>>>>> For task xyz, got file 'null' >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Tripti >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegr...@yahoo.it> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> have you seen the hadoop logs? >>>>>>>>> Could you write here how are you running the command? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Alessandro >>>>>>>>> >>>>>>>>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjin...@gmail.com> >>>>>>>>> ha scritto: >>>>>>>>> >>>>>>>>>> Yeah, I add -yj option. The result is the same. Still error here. >>>>>>>>>> >>>>>>>>>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro >>>>>>>>>> <alenegr...@yahoo.it> wrote: >>>>>>>>>>> Hi Jing, >>>>>>>>>>> I’m facing the same problem. Do you check the hadoop logs? In my >>>>>>>>>>> case I got: >>>>>>>>>>> Could not find or load main class >>>>>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster >>>>>>>>>>> Moreover I don’t know why my application hang for a long time here: >>>>>>>>>>> >>>>>>>>>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>>>>>>>>>> >>>>>>>>>>> And then continue failing with your error into the output and the >>>>>>>>>>> other into the hadoop logs. >>>>>>>>>>> >>>>>>>>>>> Did you test adding this? >>>>>>>>>>> >>>>>>>>>>> -yj >>>>>>>>>>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Alessandro >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan >>>>>>>>>>> <fanjin...@gmail.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I install the Giraph with Yarn. >>>>>>>>>>>> But when I run the program with following command: >>>>>>>>>>>> >>>>>>>>>>>> hadoop jar >>>>>>>>>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>>>>>>>>>> org.apache.giraph.GiraphRunner >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>>>>> -vip /user/j/tiny_graph.txt -vof >>>>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>>>>>>>>>> /user/j/shortestpaths -w 1 >>>>>>>>>>>> >>>>>>>>>>>> The program exited with error: >>>>>>>>>>>> >>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, >>>>>>>>>>>> Elapsed: 0.83 secs >>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >>>>>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers >>>>>>>>>>>> used: 0 >>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, >>>>>>>>>>>> Elapsed: 4.84 secs >>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >>>>>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers >>>>>>>>>>>> used: 0 >>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, >>>>>>>>>>>> Elapsed: 8.85 secs >>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >>>>>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers >>>>>>>>>>>> used: 0 >>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, >>>>>>>>>>>> Elapsed: 12.86 secs >>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >>>>>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers >>>>>>>>>>>> used: 0 >>>>>>>>>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports >>>>>>>>>>>> FAILED state, diagnostics show: Application >>>>>>>>>>>> application_1415203231170_0002 failed 2 times due to AM Container >>>>>>>>>>>> for appattempt_1415203231170_0002_000002 exited with exitCode: 1 >>>>>>>>>>>> due to: Exception from container-launch. >>>>>>>>>>>> Container id: container_1415203231170_0002_02_000001 >>>>>>>>>>>> Exit code: 1 >>>>>>>>>>>> Stack trace: ExitCodeException exitCode=1: >>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Anyone can help? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >