Hi Eli/Alexandre, I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using the steps mentioned by Alexandre, but again i am getting the same error while running the example:
Command: $HADOOP_HOME/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /giraphInput/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostics show: Application application_1386601013698_0005 failed 2 times due to AM Container for appattempt_1386601013698_0005_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) .Failing this attempt.. Failing the application. I checked the Hadoop logs as well but couldn't find anything meaningful there too. Thanks, Rohit On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman <apache.mail...@gmail.com>wrote: > Hey all, > > So. I think from more recent emails Rohit got this figured out, but, a > couple points just so they out there again: > > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop > 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687 > ticket) against Hadoop 2.0.4 and 2.0.5 alpha. > > Much better way to go is to build against Hadoop 2.2.x line using Giraph > 1.1.x's YARN profile. This features a stable API, some basic support for > security, and a concurrency fix that could bite you when launching many > workers. > > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN profile. > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do, > but riding on Mapper task instead of running the workers in YARN containers. > > > > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca < > alexandrejorgefons...@gmail.com> wrote: > >> Hello Yong, >> >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of >> hadoop-2.2.0. >> Here's what I did: >> >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git >> cd giraph >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63 >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile >> >> With that I got valid jars which allowed me to run the SimpleShortestPaths >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5: >> >> mkdir test_example >> ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0- >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar >> $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for- >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >> -vip >> giraph/input/tiny_graph.txt -vof >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for- >> hadoop-2.2.0-jar-with-dependencies.jar >> >> ########### >> Notes: >> * Wasn't able to do mvn package as it complained about not having the >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a >> matter >> of changing the main pom.xml to include those modules with the hadoop >> 2.2.0 >> target or remove those dependencies from the giraph-dist if they no longer >> apply. >> * You may get some memory limit exceeded problems for the >> ApplicationMaster. >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in >> yarn- >> site.xml or apply my patch to prevent allocation of heap size equal to >> total >> memory size, described at: >> https://issues.apache.org/jira/browse/GIRAPH-812 >> * In my case, the application didn't exit even after outputting the >> result and >> having recognized that all nodes voted to terminate. I've opened an issue >> in JIRA regarding this so if you suffer from the same, you can apply that >> patch >> too: https://issues.apache.org/jira/browse/GIRAPH-811 >> >> ########### >> Relevant JIRA issues: >> * https://issues.apache.org/jira/browse/GIRAPH-798 >> * https://issues.apache.org/jira/browse/GIRAPH-804 >> * https://issues.apache.org/jira/browse/GIRAPH-811 >> * https://issues.apache.org/jira/browse/GIRAPH-812 >> >> Best regards, >> Alexandre >> >> On Friday 06 December 2013 00:32:32 Yong Guo wrote: >> > Hi, >> > >> > I also got this error when tried to build Giraph on top of hadoop-2.2.0 >> > using similar commands. Can anyone help? >> > >> > Thanks, >> > Yong >> > >> > >> > 2013/12/6 Rohit Bhat <rohitbhat.n...@gmail.com> >> > >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the >> following 2 >> > > commands and got similar errors in both the cases: >> > > >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package >> -DskipTests" >> > > >> > > >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install >> > > package" >> > > >> > > >> > > [INFO] Checking dependency versions >> > > >> > > [ERROR] Found a problem with the direct dependency >> > > commons-httpclient:commons-httpclient of the current project >> > > >> > > Expected version is 3.0.1 >> > > >> > > Resolved version is 3.0.1 >> > > >> > > Version 3.1 was expected by artifact: >> org.apache.hadoop:hadoop-common >> > > >> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang >> > > >> > > Resolved version is 2.4 >> > > >> > > Version 2.4 was expected by artifact: >> > > commons-configuration:commons-configuration >> > > >> > > Version 2.5 was expected by artifact: >> org.apache.hadoop:hadoop-common >> > > >> > > [ERROR] Found a problem with the dependency >> > > commons-logging:commons-logging >> > > >> > > Resolved version is 1.0.4 >> > > >> > > Version 1.1.1 was expected by artifact: >> org.apache.hadoop:hadoop-common >> > > >> > > [ERROR] Found a problem with the dependency io.netty:netty >> > > >> > > Resolved version is 3.5.3.Final >> > > >> > > Version 3.6.2.Final was expected by artifacts: org.apache. >> > > >> > > hadoop:hadoop-mapreduce-client-common, org.apache. >> > > hadoop:hadoop-mapreduce-client-core >> > > >> > > [ERROR] Found a problem with the dependency >> org.apache.zookeeper:zookeeper >> > > >> > > Resolved version is 3.3.3 >> > > >> > > Version 3.4.5 was expected by artifacts: >> > > org.apache.hadoop:hadoop-common, >> > > >> > > org.apache.hadoop:hadoop-mapreduce-client-common >> > > >> > > >> > > Am i missing something? >> > > >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar >> <ameya.vilan...@gmail.com>wrote: >> > >> Yes. Use this: >> > >> >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package >> > >> >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang >> <manuellag...@gmail.com>wrote: >> > >>> I get the same error when I compile Giraph against the default >> hadoop >> > >>> version (0.20.203.0), but my project that uses Giraph uses a more >> > >>> recent hadoop version. Did you set the hadoop version via a maven >> > >>> profile >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop >> 1.0)? >> > >>> >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even a >> > >>> Yarn profile in the pom.xml. I personally haven't had success >> building >> > >>> it >> > >>> though. >> > >>> >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat >> <rohitbhat.n...@gmail.com>wrote: >> > >>>> Hi, >> > >>>> >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have >> > >>>> installed Giraph on top of it. I have been trying to >> > >>>> execute SimpleShortestPathsComputation example but i am getting the >> > >>>> error: >> > >>>> >> > >>>> "java.lang.IllegalArgumentException: >> checkLocalJobRunnerConfiguration: >> > >>>> When using LocalJobRunner, you cannot run in split master / worker >> mode >> > >>>> since there is only 1 task at a time!" >> > >>>> >> > >>>> I fixed this by adding an entry in mapred-site.xml for >> > >>>> mapreduce.jobtracker.address but now i am getting a different >> error: >> > >>>> >> > >>>> *Exception in thread "main" java.lang.IncompatibleClassChangeError: >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class >> was >> > >>>> expected* >> > >>>> >> > >>>> at >> > >>>> >> > >>>> >> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat. >> > >>>> java:43) at >> > >>>> >> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4 >> > >>>> 56) >> > >>>> >> > >>>> at >> > >>>> >> > >>>> >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter >> > >>>> .java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) >> > >>>> >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) >> > >>>> >> > >>>> at java.security.AccessController.doPrivileged(Native Method) >> > >>>> >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415) >> > >>>> >> > >>>> at >> > >>>> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati >> > >>>> on.java:1491)>>>> >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) >> > >>>> >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) >> > >>>> >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) >> > >>>> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >> > >>>> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >> > >>>> >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >> > >>>> >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > >>>> >> > >>>> at >> > >>>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja >> > >>>> va:57)>>>> >> > >>>> at >> > >>>> >> > >>>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso >> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) >> > >>>> >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212) >> > >>>> >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?* >> > >>>> >> > >>>> Thanks, >> > >>>> Rohit >> > >
