if you check out GIRAPH-750 [1], you will find a patch that fixes the problem for now. You'll have to apply the patch to trunk.
Best, Claudio [1] https://issues.apache.org/jira/browse/GIRAPH-750 On Mon, Aug 26, 2013 at 11:37 PM, Claudio Martella < claudio.marte...@gmail.com> wrote: > yeah. i tracked the problem to what i mentioned earlier. ZK is run with > the wrong jar when using -libjars. I have to figure out what's the expected > behavior though, because the logic is kind obscure in the code. > > > On Mon, Aug 26, 2013 at 11:24 PM, Kyle Orlando > <kyle.r.orla...@gmail.com>wrote: > >> Yeah, exactly. We couldn't get it to work otherwise. >> >> >> On Mon, Aug 26, 2013 at 11:00 AM, Claudio Martella < >> claudio.marte...@gmail.com> wrote: >> >>> you mean by running zookeeper independently? >>> >>> >>> On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando >>> <kyle.r.orla...@gmail.com>wrote: >>> >>>> We were also experiencing similar problems when specifying -libjars as >>>> opposed to just using a fat jar. I believe we fixed it by setting the >>>> giraph.zkList property, but this only appears to work when we list one node >>>> as a zookeeper. >>>> >>>> >>>> On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella < >>>> claudio.marte...@gmail.com> wrote: >>>> >>>>> OK, then I'm going to open an issue for that. >>>>> >>>>> >>>>> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium < >>>>> vivek.semb...@gmail.com> wrote: >>>>> >>>>>> Yes for the zookeeper problem I passed a seperate jar through >>>>>> -libjars command. If I use additional jars zookeeper fails. >>>>>> On Aug 26, 2013 2:51 PM, "Claudio Martella" < >>>>>> claudio.marte...@gmail.com> wrote: >>>>>> >>>>>>> there must be a misunderstanding. i was referring to the zookeeper >>>>>>> problem. >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium < >>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>> >>>>>>>> No. I added my files(it was just a copy of one of the example >>>>>>>> program to a different package) to the jar files of giraph. But it was >>>>>>>> still giving me classNotFoundException. Can you give me some simple >>>>>>>> example >>>>>>>> program with instructions on how to deploy it. So I can start playing >>>>>>>> with >>>>>>>> giraph and make changes to the program and learn, then start working >>>>>>>> on my >>>>>>>> project in giraph. >>>>>>>> I will be very thankful if you can help me with this. >>>>>>>> >>>>>>>> Thanking you >>>>>>>> -Vivek Sembium >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella < >>>>>>>> claudio.marte...@gmail.com> wrote: >>>>>>>> >>>>>>>>> but you were still using an additional jar added through -libjars, >>>>>>>>> right? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium < >>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> @Claudio Martella Your solution didnt work either. I basically >>>>>>>>>> tried copying the pageRankBenchmark to my own package, renamed the >>>>>>>>>> package. >>>>>>>>>> It compiles fine with giraph. But I couldnt run it even if I add >>>>>>>>>> those >>>>>>>>>> files to giraph jar before deployment. Help? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella < >>>>>>>>>> claudio.marte...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> you have this problem when you use two jars (one with giraph and >>>>>>>>>>> one with your classes) instead of a single fat-jar, correct? I >>>>>>>>>>> tracked the >>>>>>>>>>> same problem a few weeks ago, basically zookeeper is run passing >>>>>>>>>>> the wrong >>>>>>>>>>> jar. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium < >>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you for your suggestion. It worked. Its not giving class >>>>>>>>>>>> not found exception. But its giving me a new error >>>>>>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found >>>>>>>>>>>> that its unable to connect to zookeeper service. >>>>>>>>>>>> >>>>>>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable >>>>>>>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>>>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>>>>>>>>>> Caused by: java.lang.IllegalStateException: >>>>>>>>>>>> onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59) >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89) >>>>>>>>>>>> ... 7 more >>>>>>>>>>>> >>>>>>>>>>>> Immediately I ran page rank benchmark and it executed >>>>>>>>>>>> successfully both from giraph in lib directory and also from >>>>>>>>>>>> giraphs own >>>>>>>>>>>> directory. >>>>>>>>>>>> >>>>>>>>>>>> Can you give me a very simple java program(finding maximum in a >>>>>>>>>>>> graph or simple page rank program) in giraph along with its jar >>>>>>>>>>>> file and >>>>>>>>>>>> input files which I can place in my lib directory of hadoop and >>>>>>>>>>>> test if >>>>>>>>>>>> its working. And also the command to execute it. This should be >>>>>>>>>>>> added in >>>>>>>>>>>> the documentation as new comers can quickly setup giraph and >>>>>>>>>>>> concentrate on >>>>>>>>>>>> their project. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ < >>>>>>>>>>>> emre.ala...@agmlab.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It isn't asking for edge input. It says make sure you don't >>>>>>>>>>>>> need it. A warning for the case you may have forgotten to give >>>>>>>>>>>>> edge input >>>>>>>>>>>>> when you really needed. >>>>>>>>>>>>> >>>>>>>>>>>>> The cause of your error is what I'm wondering nowadays. I'm >>>>>>>>>>>>> having a similar problem. Currently I'm using a workaround: put >>>>>>>>>>>>> all the >>>>>>>>>>>>> jars (giraph-core and my module giraph-nutch) in the lib folder >>>>>>>>>>>>> of hadoop. >>>>>>>>>>>>> Then it works. But there should be a clean way of doing this. >>>>>>>>>>>>> >>>>>>>>>>>>> I should be able to say hadoop jar fat.jar ... >>>>>>>>>>>>> >>>>>>>>>>>>> Any help appreciated. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ------------------------------ >>>>>>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.semb...@gmail.com> >>>>>>>>>>>>> *Kime: *user@giraph.apache.org >>>>>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49 >>>>>>>>>>>>> *Konu: *Re: Help needed for Running my own java programs in >>>>>>>>>>>>> Giraph >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I tried with and without exporting hadoop classpath. I get the >>>>>>>>>>>>> same error. >>>>>>>>>>>>> >>>>>>>>>>>>> Here's the command that I tried >>>>>>>>>>>>> hadoop jar >>>>>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar >>>>>>>>>>>>> org.apache.giraph.GiraphRunner -libjars >>>>>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar >>>>>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute >>>>>>>>>>>>> -vif >>>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input >>>>>>>>>>>>> format specified. Ensure your InputFormat does not require one. >>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output >>>>>>>>>>>>> format specified. Ensure your OutputFormat does not require one. >>>>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute >>>>>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:634) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:212) >>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>>> Method) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:314) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>>>>> at java.lang.Class.forName(Class.java:186) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>>>>>> Method) >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>>>>> Method) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>>>>> at >>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>>>>> at >>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>>>>> ... 26 more >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I gave the graph input as JSON file. Why does it ask for edge >>>>>>>>>>>>> input file again? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Why isnt this working? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando < >>>>>>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Vivek, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for the hasty response, I assumed the quick start guide >>>>>>>>>>>>>> covered that (I guess it doesn't). What you need to do first is >>>>>>>>>>>>>> put all of >>>>>>>>>>>>>> your compiled code into a jar file. Then, you'll be utilizing >>>>>>>>>>>>>> the hadoop >>>>>>>>>>>>>> option -libjars. You also need to make sure that you set the >>>>>>>>>>>>>> hadoop >>>>>>>>>>>>>> classpath to include the giraph jar you are using as well as >>>>>>>>>>>>>> your own jar. >>>>>>>>>>>>>> I actually asked a similar question not too long ago; prior to >>>>>>>>>>>>>> this, I was >>>>>>>>>>>>>> just combining everything into one jar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Your code should pretty much follow this format, which is >>>>>>>>>>>>>> copied and paster from a previous response by Claudio: >>>>>>>>>>>>>> >>>>>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar" >>>>>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D >>>>>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass [ >>>>>>>>>>>>>> GiraphRunner option e.g. -vip -vif etc. ]* >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium < >>>>>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank You for your response. I could run all the example >>>>>>>>>>>>>>> programs. I want to place my development code in a different >>>>>>>>>>>>>>> directory >>>>>>>>>>>>>>> other than the giraph directory. I copied the >>>>>>>>>>>>>>> SimplePageRankComputation.java to a different directory, >>>>>>>>>>>>>>> compiled it(with a >>>>>>>>>>>>>>> different package name). But I couldnt replace the "org. >>>>>>>>>>>>>>> apache.giraph.examples.SimpleShortestPathsComputation" with >>>>>>>>>>>>>>> my class file. It gives me a ClassNotFoundException. Please >>>>>>>>>>>>>>> Guide me so >>>>>>>>>>>>>>> that I can run any class file with all required methods (GIven >>>>>>>>>>>>>>> the location >>>>>>>>>>>>>>> of that file). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando < >>>>>>>>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> A "Quick Start" guide was recently added to the Giraph >>>>>>>>>>>>>>>> website. Now learning Giraph is easier than ever! Here: >>>>>>>>>>>>>>>> http://giraph.apache.org/quick_start.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium < >>>>>>>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am a new user in giraph. I could run all example >>>>>>>>>>>>>>>>> programs in Giraph. Can someone please guide me on how to run >>>>>>>>>>>>>>>>> my own java >>>>>>>>>>>>>>>>> program in giraph (Assuming I have my input file in jason >>>>>>>>>>>>>>>>> format in HDFS). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>>>>>> University of Maryland >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>>>> University of Maryland >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Claudio Martella >>>>>>>>>>> claudio.marte...@gmail.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Claudio Martella >>>>>>>>> claudio.marte...@gmail.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claudio Martella >>>>>>> claudio.marte...@gmail.com >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Claudio Martella >>>>> claudio.marte...@gmail.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Kyle Orlando >>>> Computer Engineering Major >>>> University of Maryland >>>> >>> >>> >>> >>> -- >>> Claudio Martella >>> claudio.marte...@gmail.com >>> >> >> >> >> -- >> Kyle Orlando >> Computer Engineering Major >> University of Maryland >> > > > > -- > Claudio Martella > claudio.marte...@gmail.com > -- Claudio Martella claudio.marte...@gmail.com
