We were also experiencing similar problems when specifying -libjars as opposed to just using a fat jar. I believe we fixed it by setting the giraph.zkList property, but this only appears to work when we list one node as a zookeeper.
On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella < claudio.marte...@gmail.com> wrote: > OK, then I'm going to open an issue for that. > > > On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium > <vivek.semb...@gmail.com>wrote: > >> Yes for the zookeeper problem I passed a seperate jar through -libjars >> command. If I use additional jars zookeeper fails. >> On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.marte...@gmail.com> >> wrote: >> >>> there must be a misunderstanding. i was referring to the zookeeper >>> problem. >>> >>> >>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <vivek.semb...@gmail.com >>> > wrote: >>> >>>> No. I added my files(it was just a copy of one of the example program >>>> to a different package) to the jar files of giraph. But it was still giving >>>> me classNotFoundException. Can you give me some simple example program with >>>> instructions on how to deploy it. So I can start playing with giraph and >>>> make changes to the program and learn, then start working on my project in >>>> giraph. >>>> I will be very thankful if you can help me with this. >>>> >>>> Thanking you >>>> -Vivek Sembium >>>> >>>> >>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella < >>>> claudio.marte...@gmail.com> wrote: >>>> >>>>> but you were still using an additional jar added through -libjars, >>>>> right? >>>>> >>>>> >>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium < >>>>> vivek.semb...@gmail.com> wrote: >>>>> >>>>>> @Claudio Martella Your solution didnt work either. I basically tried >>>>>> copying the pageRankBenchmark to my own package, renamed the package. It >>>>>> compiles fine with giraph. But I couldnt run it even if I add those files >>>>>> to giraph jar before deployment. Help? >>>>>> >>>>>> >>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella < >>>>>> claudio.marte...@gmail.com> wrote: >>>>>> >>>>>>> you have this problem when you use two jars (one with giraph and one >>>>>>> with your classes) instead of a single fat-jar, correct? I tracked the >>>>>>> same >>>>>>> problem a few weeks ago, basically zookeeper is run passing the wrong >>>>>>> jar. >>>>>>> >>>>>>> >>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium < >>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>> >>>>>>>> Thank you for your suggestion. It worked. Its not giving class not >>>>>>>> found exception. But its giving me a new error >>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found that >>>>>>>> its unable to connect to zookeeper service. >>>>>>>> >>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable >>>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) >>>>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>>>> at >>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>>>>>> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers: >>>>>>>> Failed to connect in 10 tries! >>>>>>>> at >>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727) >>>>>>>> at >>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371) >>>>>>>> at >>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204) >>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59) >>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89) >>>>>>>> ... 7 more >>>>>>>> >>>>>>>> Immediately I ran page rank benchmark and it executed successfully >>>>>>>> both from giraph in lib directory and also from giraphs own directory. >>>>>>>> >>>>>>>> Can you give me a very simple java program(finding maximum in a >>>>>>>> graph or simple page rank program) in giraph along with its jar file >>>>>>>> and >>>>>>>> input files which I can place in my lib directory of hadoop and test >>>>>>>> if >>>>>>>> its working. And also the command to execute it. This should be added >>>>>>>> in >>>>>>>> the documentation as new comers can quickly setup giraph and >>>>>>>> concentrate on >>>>>>>> their project. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ < >>>>>>>> emre.ala...@agmlab.com> wrote: >>>>>>>> >>>>>>>>> It isn't asking for edge input. It says make sure you don't need >>>>>>>>> it. A warning for the case you may have forgotten to give edge input >>>>>>>>> when >>>>>>>>> you really needed. >>>>>>>>> >>>>>>>>> The cause of your error is what I'm wondering nowadays. I'm having >>>>>>>>> a similar problem. Currently I'm using a workaround: put all the jars >>>>>>>>> (giraph-core and my module giraph-nutch) in the lib folder of hadoop. >>>>>>>>> Then >>>>>>>>> it works. But there should be a clean way of doing this. >>>>>>>>> >>>>>>>>> I should be able to say hadoop jar fat.jar ... >>>>>>>>> >>>>>>>>> Any help appreciated. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ------------------------------ >>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.semb...@gmail.com> >>>>>>>>> *Kime: *user@giraph.apache.org >>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49 >>>>>>>>> *Konu: *Re: Help needed for Running my own java programs in Giraph >>>>>>>>> >>>>>>>>> >>>>>>>>> I tried with and without exporting hadoop classpath. I get the >>>>>>>>> same error. >>>>>>>>> >>>>>>>>> Here's the command that I tried >>>>>>>>> hadoop jar >>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar >>>>>>>>> org.apache.giraph.GiraphRunner -libjars >>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar >>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute -vif >>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3 >>>>>>>>> >>>>>>>>> >>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input >>>>>>>>> format specified. Ensure your InputFormat does not require one. >>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format >>>>>>>>> specified. Ensure your OutputFormat does not require one. >>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>> org/apache/giraph/master/DefaultMasterCompute >>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:634) >>>>>>>>> at >>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>>>>> at >>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >>>>>>>>> at >>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73) >>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>> Method) >>>>>>>>> at >>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>> at >>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:314) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>> at java.lang.Class.forName(Class.java:186) >>>>>>>>> at >>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425) >>>>>>>>> at >>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408) >>>>>>>>> at >>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200) >>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>>>>> at >>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>> Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>> org.apache.giraph.master.DefaultMasterCompute >>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>> Method) >>>>>>>>> at >>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>> at >>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>> ... 26 more >>>>>>>>> >>>>>>>>> >>>>>>>>> I gave the graph input as JSON file. Why does it ask for edge >>>>>>>>> input file again? >>>>>>>>> >>>>>>>>> >>>>>>>>> Why isnt this working? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando < >>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Vivek, >>>>>>>>>> >>>>>>>>>> Sorry for the hasty response, I assumed the quick start guide >>>>>>>>>> covered that (I guess it doesn't). What you need to do first is put >>>>>>>>>> all of >>>>>>>>>> your compiled code into a jar file. Then, you'll be utilizing the >>>>>>>>>> hadoop >>>>>>>>>> option -libjars. You also need to make sure that you set the hadoop >>>>>>>>>> classpath to include the giraph jar you are using as well as your >>>>>>>>>> own jar. >>>>>>>>>> I actually asked a similar question not too long ago; prior to this, >>>>>>>>>> I was >>>>>>>>>> just combining everything into one jar. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Your code should pretty much follow this format, which is copied >>>>>>>>>> and paster from a previous response by Claudio: >>>>>>>>>> >>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar" >>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D option >>>>>>>>>> ]* -libjars your.jar your.package.ComputationClass [ >>>>>>>>>> GiraphRunner option e.g. -vip -vif etc. ]* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium < >>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thank You for your response. I could run all the example >>>>>>>>>>> programs. I want to place my development code in a different >>>>>>>>>>> directory >>>>>>>>>>> other than the giraph directory. I copied the >>>>>>>>>>> SimplePageRankComputation.java to a different directory, compiled >>>>>>>>>>> it(with a >>>>>>>>>>> different package name). But I couldnt replace the "org.apache. >>>>>>>>>>> giraph.examples.SimpleShortestPathsComputation" with my class >>>>>>>>>>> file. It gives me a ClassNotFoundException. Please Guide me so that >>>>>>>>>>> I can >>>>>>>>>>> run any class file with all required methods (GIven the location of >>>>>>>>>>> that >>>>>>>>>>> file). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando < >>>>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> A "Quick Start" guide was recently added to the Giraph website. >>>>>>>>>>>> Now learning Giraph is easier than ever! Here: >>>>>>>>>>>> http://giraph.apache.org/quick_start.html >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium < >>>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I am a new user in giraph. I could run all example programs in >>>>>>>>>>>>> Giraph. Can someone please guide me on how to run my own java >>>>>>>>>>>>> program in >>>>>>>>>>>>> giraph (Assuming I have my input file in jason format in HDFS). >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>> University of Maryland >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Kyle Orlando >>>>>>>>>> Computer Engineering Major >>>>>>>>>> University of Maryland >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claudio Martella >>>>>>> claudio.marte...@gmail.com >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Claudio Martella >>>>> claudio.marte...@gmail.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Claudio Martella >>> claudio.marte...@gmail.com >>> >> > > > -- > Claudio Martella > claudio.marte...@gmail.com > -- Kyle Orlando Computer Engineering Major University of Maryland
