Tried the steps (one variation was 'clean package' instead of 'compile' since a dependency check was failing) provided by Alexandre and hitting the same error as Rohit. Note that I'm attempting on a cluster.
Cmdline: hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/rrdata/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/rrdata/output -w 1 -yj $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar -ca giraph.SplitMasterWorker=false -ca giraph.zkList=exotica:2181 On Mon, Dec 9, 2013 at 10:34 AM, Rohit Bhat <rohitbhat.n...@gmail.com>wrote: > Hi Alexandre, > > I looked into the logs and found this: > > 2013-12-09 12:29:08,925 INFO [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:main(426)) - Starting GitaphAM > 2013-12-09 12:29:09,727 WARN [main] util.NativeCodeLoader > (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library > for your$ > 2013-12-09 12:29:10,206 ERROR [main] yarn.GiraphApplicationMaster > (GiraphApplicationMaster.java:main(442)) - GiraphApplicationMaster caught a > t$ > java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/ > *TextOutputFormat* > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:788) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > > In gam-stderr.log i found this: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/scratch/rohit/hadoop-2.2.0/tmp/nm-local-dir/usercache/rohit/appcache/application_1386613677359_0001/filecach$ > SLF4J: Found binding in > [jar:file:/scratch/rohit/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinde$ > > I tried the fix that you mentioned of linking and adding -yj for hadoop- > mapreduce-client-core-2.2.0.jar but that did not help. HADOOP_MAPRED_HOME > is pointing to HADOOP_HOME on my setup. > > > On Mon, Dec 9, 2013 at 10:11 AM, Alexandre Fonseca < > alexandrejorgefons...@gmail.com> wrote: > >> Hi Rohit, >> >> In my case, I got 2 Shell$ExitCodeExceptions when trying to run the >> SimpleShortestPathsComputation: >> >> - Error: Could not find or load main class >> org.apache.giraph.yarn.GiraphApplicationMaster >> >> This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for- >> hadoop-2.2.0-jar-with-dependencies.jar >> >> - java.lang.NoClassDefFoundError: >> org/apache/hadoop/mapreduce/lib/output/TextOutputFormat >> >> This was solved by making sure HADOOP_MAPRED_HOME was >> pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and >> restarting >> both the nodemanager and resourcemanager. Another thing that also worked >> was >> linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client- >> core-2.2.0.jar file to the current directory and adding it to the -yj >> parameter: >> -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with- >> dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar >> >> I found the first error on: >> >> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_ >> 1386443083254_0001_01_000001/gam-stderr.log >> >> And the second error on: >> >> /home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_ >> 1386443083254_0001_01_000002/gam-stdout.log >> >> If you didn't change the default yarn property yarn.nodemanager.log-dirs, >> you >> should be able to see them on the respective directories on your side. If >> not, >> check on whatever you set that property to. >> >> Hope this helps! >> >> Alex >> >> On Monday 09 December 2013 09:50:35 Rohit Bhat wrote: >> > Hi Eli/Alexandre, >> > >> > I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using >> the >> > steps mentioned by Alexandre, but again i am getting the same error >> while >> > running the example: >> > >> > Command: >> > >> > $HADOOP_HOME/bin/hadoop jar >> > >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar >> > org.apache.giraph.GiraphRunner >> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> > -vip /giraphInput/tiny_graph.txt -vof >> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> > /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca >> > giraph.zkList=localhost:2181 -yj >> > >> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar >> > >> > ERROR yarn.GiraphYarnClient: Giraph: >> > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >> > state, diagnostics show: Application application_1386601013698_0005 >> failed >> > 2 times due to AM Container for appattempt_1386601013698_0005_000002 >> exited >> > with exitCode: 1 due to: Exception from container-launch: >> > org.apache.hadoop.util.Shell$ExitCodeException: >> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) >> > at org.apache.hadoop.util.Shell.run(Shell.java:379) >> > at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) >> > at >> > >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon >> > tainer(DefaultContainerExecutor.java:195) at >> > >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe >> > rLaunch.call(ContainerLaunch.java:283) at >> > >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe >> > rLaunch.call(ContainerLaunch.java:79) at >> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at >> > java.util.concurrent.FutureTask.run(FutureTask.java:166) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 >> > 45) at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 >> > 15) at java.lang.Thread.run(Thread.java:724) >> > >> > >> > .Failing this attempt.. Failing the application. >> > >> > I checked the Hadoop logs as well but couldn't find anything meaningful >> > there too. >> > >> > Thanks, >> > Rohit >> > >> > On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman >> <apache.mail...@gmail.com>wrote: >> > > Hey all, >> > > >> > > So. I think from more recent emails Rohit got this figured out, but, a >> > > couple points just so they out there again: >> > > >> > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop >> > > 2.0.03-alpha or (with a patch thats still floating around on >> GIRAPH-687 >> > > ticket) against Hadoop 2.0.4 and 2.0.5 alpha. >> > > >> > > Much better way to go is to build against Hadoop 2.2.x line using >> Giraph >> > > 1.1.x's YARN profile. This features a stable API, some basic support >> for >> > > security, and a concurrency fix that could bite you when launching >> many >> > > workers. >> > > >> > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need >> to >> > > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN >> > > profile. >> > > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x >> do, >> > > but riding on Mapper task instead of running the workers in YARN >> > > containers. >> > > >> > > >> > > >> > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca < >> > > >> > > alexandrejorgefons...@gmail.com> wrote: >> > >> Hello Yong, >> > >> >> > >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of >> > >> hadoop-2.2.0. >> > >> >> > >> Here's what I did: >> > >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git >> > >> cd giraph >> > >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 >> > >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile >> > >> >> > >> With that I got valid jars which allowed me to run the >> > >> SimpleShortestPaths >> > >> >> > >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5: >> > >> mkdir test_example >> > >> ln -s >> > >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0- >> > >> >> > >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar >> > >> >> > >> $HADOOP_PREFIX/bin/hadoop jar >> giraph-examples-1.1.0-SNAPSHOT-for- >> > >> >> > >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner >> > >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> > >> >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> > >> -vip >> > >> giraph/input/tiny_graph.txt -vof >> > >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> > >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false >> -ca >> > >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for- >> > >> hadoop-2.2.0-jar-with-dependencies.jar >> > >> >> > >> ########### >> > >> Notes: >> > >> * Wasn't able to do mvn package as it complained about not having the >> > >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is >> just a >> > >> matter >> > >> of changing the main pom.xml to include those modules with the hadoop >> > >> 2.2.0 >> > >> target or remove those dependencies from the giraph-dist if they no >> > >> longer >> > >> apply. >> > >> * You may get some memory limit exceeded problems for the >> > >> ApplicationMaster. >> > >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio >> parameter in >> > >> yarn- >> > >> site.xml or apply my patch to prevent allocation of heap size equal >> to >> > >> total >> > >> memory size, described at: >> > >> https://issues.apache.org/jira/browse/GIRAPH-812 >> > >> * In my case, the application didn't exit even after outputting the >> > >> result and >> > >> having recognized that all nodes voted to terminate. I've opened an >> issue >> > >> in JIRA regarding this so if you suffer from the same, you can apply >> that >> > >> patch >> > >> too: https://issues.apache.org/jira/browse/GIRAPH-811 >> > >> >> > >> ########### >> > >> Relevant JIRA issues: >> > >> * https://issues.apache.org/jira/browse/GIRAPH-798 >> > >> * https://issues.apache.org/jira/browse/GIRAPH-804 >> > >> * https://issues.apache.org/jira/browse/GIRAPH-811 >> > >> * https://issues.apache.org/jira/browse/GIRAPH-812 >> > >> >> > >> Best regards, >> > >> Alexandre >> > >> >> > >> On Friday 06 December 2013 00:32:32 Yong Guo wrote: >> > >> > Hi, >> > >> > >> > >> > I also got this error when tried to build Giraph on top of >> hadoop-2.2.0 >> > >> > using similar commands. Can anyone help? >> > >> > >> > >> > Thanks, >> > >> > Yong >> > >> > >> > >> > >> > >> > 2013/12/6 Rohit Bhat <rohitbhat.n...@gmail.com> >> > >> > >> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the >> > >> >> > >> following 2 >> > >> >> > >> > > commands and got similar errors in both the cases: >> > >> > > >> > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package >> > >> >> > >> -DskipTests" >> > >> >> > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean >> install >> > >> > > package" >> > >> > > >> > >> > > >> > >> > > [INFO] Checking dependency versions >> > >> > > >> > >> > > [ERROR] Found a problem with the direct dependency >> > >> > > commons-httpclient:commons-httpclient of the current project >> > >> > > >> > >> > > Expected version is 3.0.1 >> > >> > > >> > >> > > Resolved version is 3.0.1 >> > >> >> > >> > > Version 3.1 was expected by artifact: >> > >> org.apache.hadoop:hadoop-common >> > >> >> > >> > > [ERROR] Found a problem with the dependency >> commons-lang:commons-lang >> > >> > > >> > >> > > Resolved version is 2.4 >> > >> > > >> > >> > > Version 2.4 was expected by artifact: >> > >> > > commons-configuration:commons-configuration >> > >> > > >> > >> > > Version 2.5 was expected by artifact: >> > >> org.apache.hadoop:hadoop-common >> > >> >> > >> > > [ERROR] Found a problem with the dependency >> > >> > > commons-logging:commons-logging >> > >> > > >> > >> > > Resolved version is 1.0.4 >> > >> >> > >> > > Version 1.1.1 was expected by artifact: >> > >> org.apache.hadoop:hadoop-common >> > >> >> > >> > > [ERROR] Found a problem with the dependency io.netty:netty >> > >> > > >> > >> > > Resolved version is 3.5.3.Final >> > >> > > >> > >> > > Version 3.6.2.Final was expected by artifacts: org.apache. >> > >> > > >> > >> > > hadoop:hadoop-mapreduce-client-common, org.apache. >> > >> > > hadoop:hadoop-mapreduce-client-core >> > >> > > >> > >> > > [ERROR] Found a problem with the dependency >> > >> >> > >> org.apache.zookeeper:zookeeper >> > >> >> > >> > > Resolved version is 3.3.3 >> > >> > > >> > >> > > Version 3.4.5 was expected by artifacts: >> > >> > > org.apache.hadoop:hadoop-common, >> > >> > > >> > >> > > org.apache.hadoop:hadoop-mapreduce-client-common >> > >> > > >> > >> > > >> > >> > > Am i missing something? >> > >> > > >> > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar >> > >> >> > >> <ameya.vilan...@gmail.com>wrote: >> > >> > >> Yes. Use this: >> > >> > >> >> > >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true >> package >> > >> > >> >> > >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang >> > >> >> > >> <manuellag...@gmail.com>wrote: >> > >> > >>> I get the same error when I compile Giraph against the default >> > >> >> > >> hadoop >> > >> >> > >> > >>> version (0.20.203.0), but my project that uses Giraph uses a >> more >> > >> > >>> recent hadoop version. Did you set the hadoop version via a >> maven >> > >> > >>> profile >> > >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop >> > >> >> > >> 1.0)? >> > >> >> > >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's >> even >> > >> > >>> a >> > >> > >>> Yarn profile in the pom.xml. I personally haven't had success >> > >> >> > >> building >> > >> >> > >> > >>> it >> > >> > >>> though. >> > >> > >>> >> > >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat >> > >> >> > >> <rohitbhat.n...@gmail.com>wrote: >> > >> > >>>> Hi, >> > >> > >>>> >> > >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and >> have >> > >> > >>>> installed Giraph on top of it. I have been trying to >> > >> > >>>> execute SimpleShortestPathsComputation example but i am >> getting >> > >> > >>>> the >> > >> > >>>> error: >> > >> >> > >> > >>>> "java.lang.IllegalArgumentException: >> > >> checkLocalJobRunnerConfiguration: >> > >> > >>>> When using LocalJobRunner, you cannot run in split master / >> worker >> > >> >> > >> mode >> > >> >> > >> > >>>> since there is only 1 task at a time!" >> > >> > >>>> >> > >> > >>>> I fixed this by adding an entry in mapred-site.xml for >> > >> > >>>> mapreduce.jobtracker.address but now i am getting a different >> > >> >> > >> error: >> > >> > >>>> *Exception in thread "main" >> > >> > >>>> java.lang.IncompatibleClassChangeError: >> > >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but >> class >> > >> >> > >> was >> > >> >> > >> > >>>> expected* >> > >> > >>>> >> > >> > >>>> at >> > >> >> > >> >> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat. >> > >> >> > >> > >>>> java:43) at >> > >> >> > >> >> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4 >> > >> >> > >> > >>>> 56) >> > >> > >>>> >> > >> > >>>> at >> > >> >> > >> >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter >> > >> >> > >> > >>>> .java:342) at >> > >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) >> > >> > >>>> >> > >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) >> > >> > >>>> >> > >> > >>>> at java.security.AccessController.doPrivileged(Native Method) >> > >> > >>>> >> > >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415) >> > >> > >>>> >> > >> > >>>> at >> > >> >> > >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati >> > >> >> > >> > >>>> on.java:1491)>>>> >> > >> > >>>> >> > >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) >> > >> > >>>> >> > >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) >> > >> > >>>> >> > >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) >> > >> > >>>> >> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >> > >> > >>>> >> > >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >> > >> > >>>> >> > >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >> > >> > >>>> >> > >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> > >> > >>>> >> > >> > >>>> at >> > >> >> > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja >> > >> >> > >> > >>>> va:57)>>>> >> > >> > >>>> >> > >> > >>>> at >> > >> >> > >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso >> > >> >> > >> > >>>> rImpl.java:43) at >> java.lang.reflect.Method.invoke(Method.java:606) >> > >> > >>>> >> > >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212) >> > >> > >>>> >> > >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing >> anything?* >> > >> > >>>> >> > >> > >>>> Thanks, >> > >> > >>>> Rohit >> > >
