Adding the yj argument to specify the YARN JARs doesn't make any difference.
Here is the complete console output from my run: bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/3 -w 1 -yj /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm info from SCDynamicStore 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is: hdfs://localhost/user/rvesse/shortestpaths/3 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=mbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:8042, nodeRackName/default-rack, nodeNumContainers0 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB. 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1384523910015_0004 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the application master 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hado op/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/ share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_H OME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MA PRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduc e/lib/*} 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1384523910015 _0004/giraph-conf.xml 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exam ples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name . 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking . 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/lib 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/* 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplicationMaster launch container is populated. 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to ASM 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application application_1384523910015_0004 to ResourceManager at /0.0.0.0:8032 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1384523910015_0004 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.82 secs 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: appattempt_1384523910015_0004_000001, State: ACCEPTED, Containers used: 1 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1384523910015_0004 failed 2 times due to AM Container for appattempt_1384523910015_0004_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon tainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe rLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe rLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 18) at java.lang.Thread.run(Thread.java:695) .Failing this attempt.. Failing the application. 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Giraph job. 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 seconds. And I see the same error in the logs: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) Any more ideas? Rob From: Mohammad Islam <misla...@yahoo.com> Reply-To: <user@giraph.apache.org> Date: Thursday, 14 November 2013 20:36 To: "user@giraph.apache.org" <user@giraph.apache.org> Subject: Re: Giraph and Hadoop 2.2.0 > I tried the following command few months back. I found you are not passing > some params. > > bin/hadoop --config etc/hadoop jar > /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1 > .?.0-for-hadoop-???-jar-with-dependencies.jar org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimplePageRankVertexOpt -c > org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj > giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -of > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip /user/me/input > -op /user/me/output_35w_20G_80m_1 > > > > > > On Thursday, November 14, 2013 3:22 AM, Rob Vesse <rve...@dotnetrdf.org> > wrote: > > > It's the same command line I showed earlier, just trying to run the example > application: > > mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar > /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-example > s-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > /user/rvesse/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/rvesse/shortestpaths/2 -w 1 > > Rob > > From: Mohammad Islam <misla...@yahoo.com> > Reply-To: <user@giraph.apache.org> > Date: Thursday, 14 November 2013 05:57 > To: "user@giraph.apache.org" <user@giraph.apache.org> > Subject: Re: Giraph and Hadoop 2.2.0 > >> What are you running? PR? >> What is the command line? >> >> >> >> >> >> >> On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <rve...@dotnetrdf.org> >> wrote: >> >> >> So I fixed the previous issue I was having which I reported and submitted a >> patch as GIRAPH-795 >> >> I then went ahead with installing and configuring my external Zookeeper >> appropriately and tried running again but now the YARN application fails and >> I find the following error in the logs: >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/giraph/yarn/GiraphApplicationMaster >> Caused by: java.lang.ClassNotFoundException: >> org.apache.giraph.yarn.GiraphApplicationMaster >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> >> Anybody have idea what the cause of this one might be or how to fix it? >> >> Cheers, >> >> Rob >> >> From: Rob Vesse <rve...@dotnetrdf.org> >> Reply-To: <user@giraph.apache.org> >> Date: Wednesday, 13 November 2013 15:35 >> To: <user@giraph.apache.org> >> Subject: Re: Giraph and Hadoop 2.2.0 >> >>> Mohammad >>> >>> This got me a build that at least attempts to start to run but not I see the >>> following: >>> >>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar >>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examp >>> les-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip >>> /user/rvesse/tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>> /user/rvesse/shortestpaths/2 -w 1 >>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from >>> SCDynamicStore >>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load native-hadoop >>> library for your platform... using builtin-java classes where applicable >>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input format >>> specified. Ensure your InputFormat does not require one. >>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output format >>> specified. Ensure your OutputFormat does not require one. >>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path is: >>> hdfs://localhost/user/rvesse/shortestpaths/2 >>> Exception in thread "main" java.lang.NullPointerException >>> at >>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(Gira >>> phYarnClient.java:460) >>> at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:116) >>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39>>> ) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl >>> .java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212) >>> >>> Which indirectly implies that I have some missing configuration to run >>> Giraph on YARN. I don't think that should be a NPE and it looks to be a >>> trivial fix so I'll try and fix and send a pull request for that. >>> >>> The error message I should receive if the NPE wasn't happening is the >>> following: >>> >>> Giraph on YARN does not currentlysupport Giraph-managed ZK instances: use a >>> standalone ZooKeeper. >>> >>> What configuration options do I need to have set to point Giraph at an >>> external Zookeeper? >>> >>> Cheers, >>> >>> Rob >>> >>> From: Mohammad Islam <misla...@yahoo.com> >>> Reply-To: <user@giraph.apache.org> >>> Date: Wednesday, 13 November 2013 11:15 >>> To: "user@giraph.apache.org" <user@giraph.apache.org> >>> Subject: Re: Giraph and Hadoop 2.2.0 >>> >>>> Hi Rob, >>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 >>>> clean package -DskipTests"? >>>> Regards, >>>> Mohammad >>>> >>>> >>>> >>>> >>>> >>>> On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <rve...@dotnetrdf.org> >>>> wrote: >>>> >>>> >>>> Hi all >>>> >>>> I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0 >>>> installation. I tried the hadoop_2.0.3 profile with no luck because I get >>>> class cast errors when trying to run the examples. >>>> >>>> Is there a specific profile I should be building to support Hadoop 2.2.0 or >>>> is this not supported yet? >>>> >>>> Thanks, >>>> >>>> Rob >>>> >>>> >>>> >>>> >>>> >>>> >> >> >> >> >> >> > > > > > >