Hi all,
When running the SimplePageRankComputation Giraph example, I get the
following error in the YARN logs (and nothing else):
Could not find or load main class
org.apache.giraph.yarn.GiraphApplicationMaster
On the command line, I see:
14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
14/01/08 14:40:03 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Final output path is:
hdfs://hadoop2.j7.master:8020/user/spry/output/shortestpaths
14/01/08 14:40:04 INFO yarn.GiraphYarnClient: Running Client
14/01/08 14:40:04 INFO client.RMProxy: Connecting to ResourceManager at
hadoop2.j7.master/127.0.0.1:8050
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Got node report from ASM for,
nodeId=localhost:45454, nodeAddresslocalhost:8042,
nodeRackName/default-rack, nodeNumContainers0
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Defaulting per-task heap size
to 1024MB.
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Obtained new Application ID:
application_1389197443854_0014
14/01/08 14:40:05 INFO Configuration.deprecation: mapred.job.id is
deprecated. Instead, use mapreduce.job.id
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Set the environment for the
application master
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: Environment for AM
:{CLASSPATH=${CLASSPATH}:./*:/etc/hadoop/conf:/usr/lib/hadoop/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-mapreduce/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/01/08 14:40:05 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/01/08 14:40:05 INFO Configuration.deprecation: mapred.output.dir is
deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/01/08 14:40:07 INFO yarn.YarnUtils: Registered file in LocalResources ::
hdfs://hadoop2.j7.master:8020/user/spry/giraph_yarn_jar_cache/application_1389197443854_0014/giraph-conf.xml
14/01/08 14:40:07 INFO yarn.GiraphYarnClient: LIB JARS
:giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
14/01/08 14:40:07 INFO yarn.YarnUtils: Class path name .
14/01/08 14:40:07 INFO yarn.YarnUtils: base path checking .
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name null
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking null
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /etc/hadoop/conf
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking /etc/hadoop/conf
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name /usr/lib/hadoop/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking /usr/lib/hadoop/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-hdfs/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-hdfs/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-yarn/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-yarn/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-yarn/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-yarn/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/lib/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce/lib
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/.//*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce/./
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/share/java/mysql-connector-java-5.1.17.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/share/java/mysql-connector-java-5.1.17.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/share/java/mysql-connector-java.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/share/java/mysql-connector-java.jar
14/01/08 14:40:08 INFO yarn.YarnUtils: Class path name
/usr/lib/hadoop-mapreduce/*
14/01/08 14:40:08 INFO yarn.YarnUtils: base path checking
/usr/lib/hadoop-mapreduce
14/01/08 14:40:08 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
for GiraphApplicationMaster launch container is populated.
14/01/08 14:40:08 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/01/08 14:40:09 INFO impl.YarnClientImpl: Submitted application
application_1389197443854_0014 to ResourceManager at hadoop2.j7.master/
127.0.0.1:8050
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1389197443854_0014
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/01/08 14:40:09 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.96
secs
14/01/08 14:40:09 INFO yarn.GiraphYarnClient:
appattempt_1389197443854_0014_000001, State: ACCEPTED, Containers used: 1
14/01/08 14:40:13 ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1389197443854_0014 failed
2 times due to AM Container for appattempt_1389197443854_0014_000002 exited
with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
.Failing this attempt.. Failing the application.
14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
14/01/08 14:40:13 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 4 seconds.
Googling led me to this mailing list and some suggestions to use the -yj
flag to include the giraph-core and giraph-example jar files, but that did
not help. I was unable to find any posts that said they were able to
resolve the issue (and included what they ended up trying). Does anyone see
anything wrong with using this command:
hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dgiraph.zkList="localhost:2181"
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/spry/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/spry/output/shortestpaths -w 1 -yj
giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar,giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
For background:
I compiled Giraph with this command, and everything built successfully
except "Apache Giraph Distribution" which it doesn't seem like I need:
mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests clean package
I am running with the following components:
Single node cluster
Giraph 1.1
Hadoop 2.2.0 (Hortonworks)
Java 1.7.0_45
Any help is appreciated. Thanks in advance!
Kristen Hardwick
