Hi Rohit,
In my case, I got 2 Shell$ExitCodeExceptions when trying to run the
SimpleShortestPathsComputation:
- Error: Could not find or load main class
org.apache.giraph.yarn.GiraphApplicationMaster
This was solved by adding -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar
- java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/output/TextOutputFormat
This was solved by making sure HADOOP_MAPRED_HOME was
pointing to the same directory as HADOOP_PREFIX or HADOOP_HOME and restarting
both the nodemanager and resourcemanager. Another thing that also worked was
linking the $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-
core-2.2.0.jar file to the current directory and adding it to the -yj
parameter:
-yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
dependencies.jar,hadoop-mapreduce-client-core-2.2.0.jar
I found the first error on:
/home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0001/container_
1386443083254_0001_01_000001/gam-stderr.log
And the second error on:
/home/alex/Programs/hadoop-2.2.0/logs/userlogs/application_1386443083254_0002/container_
1386443083254_0001_01_000002/gam-stdout.log
If you didn't change the default yarn property yarn.nodemanager.log-dirs, you
should be able to see them on the respective directories on your side. If not,
check on whatever you set that property to.
Hope this helps!
Alex
On Monday 09 December 2013 09:50:35 Rohit Bhat wrote:
> Hi Eli/Alexandre,
>
> I have tried with Giraph-1.1.0-SNAPSHOT and hadoop-2.2.0 as well using the
> steps mentioned by Alexandre, but again i am getting the same error while
> running the example:
>
> Command:
>
> $HADOOP_HOME/bin/hadoop jar
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /giraphInput/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> giraph.zkList=localhost:2181 -yj
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
>
> ERROR yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
> state, diagnostics show: Application application_1386601013698_0005 failed
> 2 times due to AM Container for appattempt_1386601013698_0005_000002 exited
> with exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchCon
> tainer(DefaultContainerExecutor.java:195) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> rLaunch.call(ContainerLaunch.java:283) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Containe
> rLaunch.call(ContainerLaunch.java:79) at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 45) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15) at java.lang.Thread.run(Thread.java:724)
>
>
> .Failing this attempt.. Failing the application.
>
> I checked the Hadoop logs as well but couldn't find anything meaningful
> there too.
>
> Thanks,
> Rohit
>
> On Sun, Dec 8, 2013 at 12:02 PM, Eli Reisman
<apache.mail...@gmail.com>wrote:
> > Hey all,
> >
> > So. I think from more recent emails Rohit got this figured out, but, a
> > couple points just so they out there again:
> >
> > Giraph 1.0.x can be built on a "pure YARN" profile with Hadoop
> > 2.0.03-alpha or (with a patch thats still floating around on GIRAPH-687
> > ticket) against Hadoop 2.0.4 and 2.0.5 alpha.
> >
> > Much better way to go is to build against Hadoop 2.2.x line using Giraph
> > 1.1.x's YARN profile. This features a stable API, some basic support for
> > security, and a concurrency fix that could bite you when launching many
> > workers.
> >
> > Finally, you _can_ run Giraph on MRv2 on YARN in which case you need to
> > build against a Hadoop 2.x Giraph Maven profile but _not_ the YARN
> > profile.
> > This runs on top of MapReduce just as Giraph on Hadoop 1.x or 0.20.x do,
> > but riding on Mapper task instead of running the workers in YARN
> > containers.
> >
> >
> >
> > On Sat, Dec 7, 2013 at 8:53 PM, Alexandre Fonseca <
> >
> > alexandrejorgefons...@gmail.com> wrote:
> >> Hello Yong,
> >>
> >> I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of
> >> hadoop-2.2.0.
> >>
> >> Here's what I did:
> >> git clone https://git-wip-us.apache.org/repos/asf/giraph.git
> >> cd giraph
> >> git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
> >> mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -DskipTests compile
> >>
> >> With that I got valid jars which allowed me to run the
> >> SimpleShortestPaths
> >>
> >> example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:
> >> mkdir test_example
> >> ln -s
> >> $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1.0-
> >>
> >> SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-
> >>
> >> hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner
> >> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> >> -vip
> >> giraph/input/tiny_graph.txt -vof
> >> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >> giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false -ca
> >> giraph.zkList=localhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
> >> hadoop-2.2.0-jar-with-dependencies.jar
> >>
> >> ###########
> >> Notes:
> >> * Wasn't able to do mvn package as it complained about not having the
> >> dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a
> >> matter
> >> of changing the main pom.xml to include those modules with the hadoop
> >> 2.2.0
> >> target or remove those dependencies from the giraph-dist if they no
> >> longer
> >> apply.
> >> * You may get some memory limit exceeded problems for the
> >> ApplicationMaster.
> >> To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter in
> >> yarn-
> >> site.xml or apply my patch to prevent allocation of heap size equal to
> >> total
> >> memory size, described at:
> >> https://issues.apache.org/jira/browse/GIRAPH-812
> >> * In my case, the application didn't exit even after outputting the
> >> result and
> >> having recognized that all nodes voted to terminate. I've opened an issue
> >> in JIRA regarding this so if you suffer from the same, you can apply that
> >> patch
> >> too: https://issues.apache.org/jira/browse/GIRAPH-811
> >>
> >> ###########
> >> Relevant JIRA issues:
> >> * https://issues.apache.org/jira/browse/GIRAPH-798
> >> * https://issues.apache.org/jira/browse/GIRAPH-804
> >> * https://issues.apache.org/jira/browse/GIRAPH-811
> >> * https://issues.apache.org/jira/browse/GIRAPH-812
> >>
> >> Best regards,
> >> Alexandre
> >>
> >> On Friday 06 December 2013 00:32:32 Yong Guo wrote:
> >> > Hi,
> >> >
> >> > I also got this error when tried to build Giraph on top of hadoop-2.2.0
> >> > using similar commands. Can anyone help?
> >> >
> >> > Thanks,
> >> > Yong
> >> >
> >> >
> >> > 2013/12/6 Rohit Bhat <rohitbhat.n...@gmail.com>
> >> >
> >> > > I already have yarn (hadoop_2.2.0) on my machine. I tried the
> >>
> >> following 2
> >>
> >> > > commands and got similar errors in both the cases:
> >> > >
> >> > > 1. "mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package
> >>
> >> -DskipTests"
> >>
> >> > > 2. "mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install
> >> > > package"
> >> > >
> >> > >
> >> > > [INFO] Checking dependency versions
> >> > >
> >> > > [ERROR] Found a problem with the direct dependency
> >> > > commons-httpclient:commons-httpclient of the current project
> >> > >
> >> > > Expected version is 3.0.1
> >> > >
> >> > > Resolved version is 3.0.1
> >>
> >> > > Version 3.1 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency commons-lang:commons-lang
> >> > >
> >> > > Resolved version is 2.4
> >> > >
> >> > > Version 2.4 was expected by artifact:
> >> > > commons-configuration:commons-configuration
> >> > >
> >> > > Version 2.5 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency
> >> > > commons-logging:commons-logging
> >> > >
> >> > > Resolved version is 1.0.4
> >>
> >> > > Version 1.1.1 was expected by artifact:
> >> org.apache.hadoop:hadoop-common
> >>
> >> > > [ERROR] Found a problem with the dependency io.netty:netty
> >> > >
> >> > > Resolved version is 3.5.3.Final
> >> > >
> >> > > Version 3.6.2.Final was expected by artifacts: org.apache.
> >> > >
> >> > > hadoop:hadoop-mapreduce-client-common, org.apache.
> >> > > hadoop:hadoop-mapreduce-client-core
> >> > >
> >> > > [ERROR] Found a problem with the dependency
> >>
> >> org.apache.zookeeper:zookeeper
> >>
> >> > > Resolved version is 3.3.3
> >> > >
> >> > > Version 3.4.5 was expected by artifacts:
> >> > > org.apache.hadoop:hadoop-common,
> >> > >
> >> > > org.apache.hadoop:hadoop-mapreduce-client-common
> >> > >
> >> > >
> >> > > Am i missing something?
> >> > >
> >> > > On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar
> >>
> >> <ameya.vilan...@gmail.com>wrote:
> >> > >> Yes. Use this:
> >> > >>
> >> > >> mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package
> >> > >>
> >> > >> On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang
> >>
> >> <manuellag...@gmail.com>wrote:
> >> > >>> I get the same error when I compile Giraph against the default
> >>
> >> hadoop
> >>
> >> > >>> version (0.20.203.0), but my project that uses Giraph uses a more
> >> > >>> recent hadoop version. Did you set the hadoop version via a maven
> >> > >>> profile
> >> > >>> when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop
> >>
> >> 1.0)?
> >>
> >> > >>> Presumably, there's a way to compile Giraph for Yarn. There's even
> >> > >>> a
> >> > >>> Yarn profile in the pom.xml. I personally haven't had success
> >>
> >> building
> >>
> >> > >>> it
> >> > >>> though.
> >> > >>>
> >> > >>> On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat
> >>
> >> <rohitbhat.n...@gmail.com>wrote:
> >> > >>>> Hi,
> >> > >>>>
> >> > >>>> I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have
> >> > >>>> installed Giraph on top of it. I have been trying to
> >> > >>>> execute SimpleShortestPathsComputation example but i am getting
> >> > >>>> the
> >> > >>>> error:
> >>
> >> > >>>> "java.lang.IllegalArgumentException:
> >> checkLocalJobRunnerConfiguration:
> >> > >>>> When using LocalJobRunner, you cannot run in split master / worker
> >>
> >> mode
> >>
> >> > >>>> since there is only 1 task at a time!"
> >> > >>>>
> >> > >>>> I fixed this by adding an entry in mapred-site.xml for
> >> > >>>> mapreduce.jobtracker.address but now i am getting a different
> >>
> >> error:
> >> > >>>> *Exception in thread "main"
> >> > >>>> java.lang.IncompatibleClassChangeError:
> >> > >>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class
> >>
> >> was
> >>
> >> > >>>> expected*
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.
> >>
> >> > >>>> java:43) at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:4
> >>
> >> > >>>> 56)
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter
> >>
> >> > >>>> .java:342) at
> >> > >>>> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> >> > >>>>
> >> > >>>> at java.security.AccessController.doPrivileged(Native Method)
> >> > >>>>
> >> > >>>> at javax.security.auth.Subject.doAs(Subject.java:415)
> >> > >>>>
> >> > >>>> at
> >>
> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> >>
> >> > >>>> on.java:1491)>>>>
> >> > >>>>
> >> > >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> >> > >>>>
> >> > >>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
> >> > >>>>
> >> > >>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >> > >>>>
> >> > >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> >> > >>>>
> >> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > >>>>
> >> > >>>> at
> >>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> >>
> >> > >>>> va:57)>>>>
> >> > >>>>
> >> > >>>> at
> >>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> >>
> >> > >>>> rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
> >> > >>>>
> >> > >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >> > >>>>
> >> > >>>> *Is Giraph compatible with yarn? If yes, am i missing anything?*
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>> Rohit
