Hi Tripti,
let me add one info, As you can see from this thread of email:
http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3cc3f75a98-bde8-4205-971f-1f7241e6e...@yahoo.it%3E
I had some issue while compiling for Hadoop 2.5.1 could be that into the
process I loose something?
Thanks again,
Alessandro
Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegr...@yahoo.it>
ha scritto:
> Hi Tripti,
> yes I run in this way:
>
>
> $HADOOP_INSTALL/bin/hadoop jar
> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip
> /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
> -op /shortestpaths -w 1 -yj
> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
> And this is the complete trace:
>
> 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from
> SCDynamicStore
> 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://hadoop-master:9000/shortestpaths
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client
> 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for,
> nodeId=10.61.137.28:59677, nodeAddress 10.61.137.28:8042, nodeRackName
> /default-rack, nodeNumContainers 0
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
> to 1024MB.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID:
> application_1415264041937_0005
> 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is
> deprecated. Instead, use mapreduce.job.id
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the
> application master
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM
> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources ::
> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415264041937_0005/giraph-conf.xml
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS
> :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/common
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /opt/yarn/hadoop-2.5.1/share/myLib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /Users/ale/extprj/giraph/myGiraph/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /Users/ale/extprj/giraph/myGiraph
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /Users/ale/extprj/giraph/myGiraph/lib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /Users/ale/extprj/giraph/myGiraph/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> /contrib/capacity-scheduler/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
> /contrib/capacity-scheduler
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
> for GiraphApplicationMaster launch container is populated.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application
> application_1415264041937_0005
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submission
> :application_1415264041937_0005
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
> container request was submitted to ResourceManager for job: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation
> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83 secs
> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient:
> appattempt_1415264041937_0005_000001, State: ACCEPTED, Containers used: 1
> 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1415264041937_0005 failed 2
> times due to AM Container for appattempt_1415264041937_0005_000002 exited
> with exitCode: 1 due to: Exception from container-launch: ExitCodeException
> exitCode=1:
> ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> Container exited with a non-zero exit code 1
> .Failing this attempt.. Failing the application.
> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
> cache directory for Giraph job.
> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
> running time: 0 minutes, 3 seconds.
> ale:~/extprj/giraph/myGiraph />yarn logs -applicationId
> application_1415264041937_0005
>
> As you can see there is this piece:
>
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS
> :/Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
>
> But I continue to receive the error:
>
> Thanks,
> Alessandro
>
>
> Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tri...@yahoo-inc.com> ha
> scritto:
>
>> For GiraphApplicationMaster not found, the simple thing that worked for me
>> was adding a -yj parameter to my command.
>> (hadoop jar {jar-file-path} {path to giraph runner} {computation-class}
>> {bunch of parameters} -yj {jar-file-name})
>> If u look at job logs,
>> Do u see a log statement defining "LIB JARS"?
>>
>> Sent from my iPhone
>>
>> On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegr...@yahoo.it> wrote:
>>
>>> Hi,
>>> yea my logs shows that error. I follow this example:
>>>
>>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3c53f4c689.5060...@web.de%3E
>>>
>>> without success.
>>> Then I try to add all jars I can from giraph adding the following lines
>>> into the hadoop-env.sh:
>>>
>>> export
>>> MYLIB=/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>> export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$MYLIB
>>>
>>>
>>> when I run the application now I see this:
>>>
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
>>> /opt/yarn/hadoop-2.5.1/share/myLib/*.jar
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
>>> /opt/yarn/hadoop-2.5.1/share/myLib
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
>>> /Users/ale/extprj/giraph/myGiraph/*.jar
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
>>> /Users/ale/extprj/giraph/myGiraph
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
>>> /Users/ale/extprj/giraph/myGiraph/lib/*.jar
>>> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking
>>> /Users/ale/extprj/giraph/myGiraph/lib
>>>
>>>
>>> But again the same error. What I think is that there is no problem in the
>>> library but while is starting up.
>>> I saw this piece in the logs:
>>>
>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
>>> for GiraphApplicationMaster launch container is populated.
>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
>>> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application
>>> application_1415264041937_0005
>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after
>>> submission :application_1415264041937_0005
>>> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>> container request was submitted to ResourceManager for job: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,83
>>> secs
>>>
>>>
>>> That show that it is able to load GiraphApplicationMaster but fail during
>>> the startup.
>>> Where can I find more logs?
>>>
>>> Thanks,
>>> Alessandro
>>>
>>>
>>> Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh <tri...@yahoo-inc.com>
>>> ha scritto:
>>>
>>>> Can u run the following command to find container logs and post it here:
>>>>
>>>> yarn logs -applicationId application_1415203231170_0002
>>>>
>>>> If u find this error:
>>>> Could not find or load main class
>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>> It may just be due to non-availablity of the yarn jars
>>>> Thanks,
>>>> Tripti
>>>> Tech Yahoo
>>>>
>>>> From: Jing Fan <fanjin...@gmail.com>
>>>> Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>> Date: Thursday, November 6, 2014 at 1:56 AM
>>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
>>>> Subject: Re: Help with Giraph on Yarn
>>>>
>>>> Then do you know to solve this error?
>>>>
>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
>>>> state, diagnostics show: Application application_1415203231170_0002 failed
>>>> 2 times due to AM Container for appattempt_1415203231170_0002_000002
>>>> exited with exitCode: 1 due to: Exception from container-launch.
>>>> Container id: container_1415203231170_0002_02_000001
>>>> Exit code: 1
>>>> Stack trace: ExitCodeException exitCode=1:
>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>> at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>> at
>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro <alenegr...@yahoo.it>
>>>> wrote:
>>>>> Hi,
>>>>> in the mean time I found the reason why it hangs for long time here:
>>>>>
>>>>> base path checking .
>>>>>
>>>>>
>>>>> It scan all the directory tree starting from current directory, if you
>>>>> run it into your home (and you have a lot of files and directories) it
>>>>> takes ages to complete.
>>>>>
>>>>> I continue to got the same error.
>>>>>
>>>>> Regards,
>>>>> Alessandro
>>>>>
>>>>> Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjin...@gmail.com> ha
>>>>> scritto:
>>>>>
>>>>>> I run with:
>>>>>>
>>>>>> hadoop jar
>>>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner
>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>> -vip /user/jing/tiny_graph.txt -vof
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>> /user/jing/shortestpaths -w 1 -yj
>>>>>> giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegr...@yahoo.it>
>>>>>> wrote:
>>>>>>> This is my complete logs with the command I run:
>>>>>>>
>>>>>>>
>>>>>>> ale:~ />$HADOOP_INSTALL/bin/hadoop jar
>>>>>>> /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /tiny_graph.txt -vof
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>> /shortestpaths -w 1
>>>>>>> 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from
>>>>>>> SCDynamicStore
>>>>>>> 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load
>>>>>>> native-hadoop library for your platform... using builtin-java classes
>>>>>>> where applicable
>>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format
>>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>> 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is:
>>>>>>> hdfs://hadoop-master:9000/shortestpaths
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client
>>>>>>> 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at
>>>>>>> /0.0.0.0:8032
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM
>>>>>>> for, nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042,
>>>>>>> nodeRackName /default-rack, nodeNumContainers 0
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>>>>>>> size to 1024MB.
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application
>>>>>>> ID: application_1415189069846_0004
>>>>>>> 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is
>>>>>>> deprecated. Instead, use mapreduce.job.id
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for
>>>>>>> the application master
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM
>>>>>>> :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in
>>>>>>> LocalResources ::
>>>>>>> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml
>>>>>>> 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS :
>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name .
>>>>>>> 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking .
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/etc/hadoop
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/common
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/yarn
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/*
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name
>>>>>>> /contrib/capacity-scheduler/*.jar
>>>>>>> 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking
>>>>>>> /contrib/capacity-scheduler
>>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient:
>>>>>>> ApplicationSumbissionContext for GiraphApplicationMaster launch
>>>>>>> container is populated.
>>>>>>> 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to
>>>>>>> ASM
>>>>>>> 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application
>>>>>>> application_1415189069846_0004
>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after
>>>>>>> submission :application_1415189069846_0004
>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
>>>>>>> container request was submitted to ResourceManager for job: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation
>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed:
>>>>>>> 0,82 secs
>>>>>>> 14/11/05 14:42:57 INFO yarn.GiraphYarnClient:
>>>>>>> appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used:
>>>>>>> 1
>>>>>>> 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports
>>>>>>> FAILED state, diagnostics show: Application
>>>>>>> application_1415189069846_0004 failed 2 times due to AM Container for
>>>>>>> appattempt_1415189069846_0004_000002 exited with exitCode: 1 due to:
>>>>>>> Exception from container-launch: ExitCodeException exitCode=1:
>>>>>>> ExitCodeException exitCode=1:
>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>> at
>>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>
>>>>>>>
>>>>>>> Container exited with a non-zero exit code 1
>>>>>>> .Failing this attempt.. Failing the application.
>>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS
>>>>>>> distributed cache directory for Giraph job.
>>>>>>> 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph:
>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED,
>>>>>>> total running time: 0 minutes, 3 seconds.
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh
>>>>>>> <tri...@yahoo-inc.com> ha scritto:
>>>>>>>
>>>>>>>> I faced the similar problem where the workflow just hangs.
>>>>>>>> Can u see in ur logs if there is a task with id 0.
>>>>>>>> I made a local fix and I could give u the patch if u see a similar
>>>>>>>> problem and ur workflow is unable to find the zookeeper manager file
>>>>>>>> and repeats the following message:
>>>>>>>> For task xyz, got file 'null'
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tripti
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegr...@yahoo.it>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> have you seen the hadoop logs?
>>>>>>>>> Could you write here how are you running the command?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Alessandro
>>>>>>>>>
>>>>>>>>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjin...@gmail.com>
>>>>>>>>> ha scritto:
>>>>>>>>>
>>>>>>>>>> Yeah, I add -yj option. The result is the same. Still error here.
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro
>>>>>>>>>> <alenegr...@yahoo.it> wrote:
>>>>>>>>>>> Hi Jing,
>>>>>>>>>>> I’m facing the same problem. Do you check the hadoop logs? In my
>>>>>>>>>>> case I got:
>>>>>>>>>>> Could not find or load main class
>>>>>>>>>>> org.apache.giraph.yarn.GiraphApplicationMaster
>>>>>>>>>>> Moreover I don’t know why my application hang for a long time here:
>>>>>>>>>>>
>>>>>>>>>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking .
>>>>>>>>>>>
>>>>>>>>>>> And then continue failing with your error into the output and the
>>>>>>>>>>> other into the hadoop logs.
>>>>>>>>>>>
>>>>>>>>>>> Did you test adding this?
>>>>>>>>>>>
>>>>>>>>>>> -yj
>>>>>>>>>>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Alessandro
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan
>>>>>>>>>>> <fanjin...@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I install the Giraph with Yarn.
>>>>>>>>>>>> But when I run the program with following command:
>>>>>>>>>>>>
>>>>>>>>>>>> hadoop jar
>>>>>>>>>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar
>>>>>>>>>>>> org.apache.giraph.GiraphRunner
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>>>>>>> -vip /user/j/tiny_graph.txt -vof
>>>>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>>>>>>>>>>> /user/j/shortestpaths -w 1
>>>>>>>>>>>>
>>>>>>>>>>>> The program exited with error:
>>>>>>>>>>>>
>>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>>>>>>>>>>> Elapsed: 0.83 secs
>>>>>>>>>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient:
>>>>>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers
>>>>>>>>>>>> used: 0
>>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>>>>>>>>>>> Elapsed: 4.84 secs
>>>>>>>>>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient:
>>>>>>>>>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers
>>>>>>>>>>>> used: 0
>>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>>>>>>>>>>> Elapsed: 8.85 secs
>>>>>>>>>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient:
>>>>>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers
>>>>>>>>>>>> used: 0
>>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph:
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>>>>>>>>>>> Elapsed: 12.86 secs
>>>>>>>>>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient:
>>>>>>>>>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers
>>>>>>>>>>>> used: 0
>>>>>>>>>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph:
>>>>>>>>>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports
>>>>>>>>>>>> FAILED state, diagnostics show: Application
>>>>>>>>>>>> application_1415203231170_0002 failed 2 times due to AM Container
>>>>>>>>>>>> for appattempt_1415203231170_0002_000002 exited with exitCode: 1
>>>>>>>>>>>> due to: Exception from container-launch.
>>>>>>>>>>>> Container id: container_1415203231170_0002_02_000001
>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>> Stack trace: ExitCodeException exitCode=1:
>>>>>>>>>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>>>>>>>>>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone can help?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>
