you mean by running zookeeper independently?
On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando <kyle.r.orla...@gmail.com>wrote: > We were also experiencing similar problems when specifying -libjars as > opposed to just using a fat jar. I believe we fixed it by setting the > giraph.zkList property, but this only appears to work when we list one node > as a zookeeper. > > > On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella < > claudio.marte...@gmail.com> wrote: > >> OK, then I'm going to open an issue for that. >> >> >> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium >> <vivek.semb...@gmail.com>wrote: >> >>> Yes for the zookeeper problem I passed a seperate jar through -libjars >>> command. If I use additional jars zookeeper fails. >>> On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.marte...@gmail.com> >>> wrote: >>> >>>> there must be a misunderstanding. i was referring to the zookeeper >>>> problem. >>>> >>>> >>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium < >>>> vivek.semb...@gmail.com> wrote: >>>> >>>>> No. I added my files(it was just a copy of one of the example program >>>>> to a different package) to the jar files of giraph. But it was still >>>>> giving >>>>> me classNotFoundException. Can you give me some simple example program >>>>> with >>>>> instructions on how to deploy it. So I can start playing with giraph and >>>>> make changes to the program and learn, then start working on my project in >>>>> giraph. >>>>> I will be very thankful if you can help me with this. >>>>> >>>>> Thanking you >>>>> -Vivek Sembium >>>>> >>>>> >>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella < >>>>> claudio.marte...@gmail.com> wrote: >>>>> >>>>>> but you were still using an additional jar added through -libjars, >>>>>> right? >>>>>> >>>>>> >>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium < >>>>>> vivek.semb...@gmail.com> wrote: >>>>>> >>>>>>> @Claudio Martella Your solution didnt work either. I basically tried >>>>>>> copying the pageRankBenchmark to my own package, renamed the package. It >>>>>>> compiles fine with giraph. But I couldnt run it even if I add those >>>>>>> files >>>>>>> to giraph jar before deployment. Help? >>>>>>> >>>>>>> >>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella < >>>>>>> claudio.marte...@gmail.com> wrote: >>>>>>> >>>>>>>> you have this problem when you use two jars (one with giraph and >>>>>>>> one with your classes) instead of a single fat-jar, correct? I tracked >>>>>>>> the >>>>>>>> same problem a few weeks ago, basically zookeeper is run passing the >>>>>>>> wrong >>>>>>>> jar. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium < >>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thank you for your suggestion. It worked. Its not giving class not >>>>>>>>> found exception. But its giving me a new error >>>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found that >>>>>>>>> its unable to connect to zookeeper service. >>>>>>>>> >>>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable >>>>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>>>>> at >>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>>>>>>> Caused by: java.lang.IllegalStateException: >>>>>>>>> onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>>>>> at >>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727) >>>>>>>>> at >>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371) >>>>>>>>> at >>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204) >>>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59) >>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89) >>>>>>>>> ... 7 more >>>>>>>>> >>>>>>>>> Immediately I ran page rank benchmark and it executed successfully >>>>>>>>> both from giraph in lib directory and also from giraphs own directory. >>>>>>>>> >>>>>>>>> Can you give me a very simple java program(finding maximum in a >>>>>>>>> graph or simple page rank program) in giraph along with its jar file >>>>>>>>> and >>>>>>>>> input files which I can place in my lib directory of hadoop and test >>>>>>>>> if >>>>>>>>> its working. And also the command to execute it. This should be added >>>>>>>>> in >>>>>>>>> the documentation as new comers can quickly setup giraph and >>>>>>>>> concentrate on >>>>>>>>> their project. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ < >>>>>>>>> emre.ala...@agmlab.com> wrote: >>>>>>>>> >>>>>>>>>> It isn't asking for edge input. It says make sure you don't need >>>>>>>>>> it. A warning for the case you may have forgotten to give edge input >>>>>>>>>> when >>>>>>>>>> you really needed. >>>>>>>>>> >>>>>>>>>> The cause of your error is what I'm wondering nowadays. I'm >>>>>>>>>> having a similar problem. Currently I'm using a workaround: put all >>>>>>>>>> the >>>>>>>>>> jars (giraph-core and my module giraph-nutch) in the lib folder of >>>>>>>>>> hadoop. >>>>>>>>>> Then it works. But there should be a clean way of doing this. >>>>>>>>>> >>>>>>>>>> I should be able to say hadoop jar fat.jar ... >>>>>>>>>> >>>>>>>>>> Any help appreciated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ------------------------------ >>>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.semb...@gmail.com> >>>>>>>>>> *Kime: *user@giraph.apache.org >>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49 >>>>>>>>>> *Konu: *Re: Help needed for Running my own java programs in >>>>>>>>>> Giraph >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I tried with and without exporting hadoop classpath. I get the >>>>>>>>>> same error. >>>>>>>>>> >>>>>>>>>> Here's the command that I tried >>>>>>>>>> hadoop jar >>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar >>>>>>>>>> org.apache.giraph.GiraphRunner -libjars >>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar >>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute -vif >>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input >>>>>>>>>> format specified. Ensure your InputFormat does not require one. >>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format >>>>>>>>>> specified. Ensure your OutputFormat does not require one. >>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute >>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:634) >>>>>>>>>> at >>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>>>>>> at >>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >>>>>>>>>> at >>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73) >>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>> Method) >>>>>>>>>> at >>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>> at >>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:314) >>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>>> at java.lang.Class.forName(Class.java:186) >>>>>>>>>> at >>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425) >>>>>>>>>> at >>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408) >>>>>>>>>> at >>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200) >>>>>>>>>> at >>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>>>>>> at >>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>>>>> Method) >>>>>>>>>> at >>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>>>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute >>>>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>>>>> at java.security.AccessController.doPrivileged(Native >>>>>>>>>> Method) >>>>>>>>>> at >>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>>>>> at >>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>>>>> ... 26 more >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I gave the graph input as JSON file. Why does it ask for edge >>>>>>>>>> input file again? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Why isnt this working? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando < >>>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Vivek, >>>>>>>>>>> >>>>>>>>>>> Sorry for the hasty response, I assumed the quick start guide >>>>>>>>>>> covered that (I guess it doesn't). What you need to do first is put >>>>>>>>>>> all of >>>>>>>>>>> your compiled code into a jar file. Then, you'll be utilizing the >>>>>>>>>>> hadoop >>>>>>>>>>> option -libjars. You also need to make sure that you set the hadoop >>>>>>>>>>> classpath to include the giraph jar you are using as well as your >>>>>>>>>>> own jar. >>>>>>>>>>> I actually asked a similar question not too long ago; prior to >>>>>>>>>>> this, I was >>>>>>>>>>> just combining everything into one jar. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Your code should pretty much follow this format, which is copied >>>>>>>>>>> and paster from a previous response by Claudio: >>>>>>>>>>> >>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar" >>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D >>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass [ >>>>>>>>>>> GiraphRunner option e.g. -vip -vif etc. ]* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium < >>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank You for your response. I could run all the example >>>>>>>>>>>> programs. I want to place my development code in a different >>>>>>>>>>>> directory >>>>>>>>>>>> other than the giraph directory. I copied the >>>>>>>>>>>> SimplePageRankComputation.java to a different directory, compiled >>>>>>>>>>>> it(with a >>>>>>>>>>>> different package name). But I couldnt replace the "org.apache. >>>>>>>>>>>> giraph.examples.SimpleShortestPathsComputation" with my class >>>>>>>>>>>> file. It gives me a ClassNotFoundException. Please Guide me so >>>>>>>>>>>> that I can >>>>>>>>>>>> run any class file with all required methods (GIven the location >>>>>>>>>>>> of that >>>>>>>>>>>> file). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando < >>>>>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> A "Quick Start" guide was recently added to the Giraph >>>>>>>>>>>>> website. Now learning Giraph is easier than ever! Here: >>>>>>>>>>>>> http://giraph.apache.org/quick_start.html >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium < >>>>>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I am a new user in giraph. I could run all example programs >>>>>>>>>>>>>> in Giraph. Can someone please guide me on how to run my own java >>>>>>>>>>>>>> program in >>>>>>>>>>>>>> giraph (Assuming I have my input file in jason format in HDFS). >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Kyle Orlando >>>>>>>>>>>>> Computer Engineering Major >>>>>>>>>>>>> University of Maryland >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Kyle Orlando >>>>>>>>>>> Computer Engineering Major >>>>>>>>>>> University of Maryland >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claudio Martella >>>>>>>> claudio.marte...@gmail.com >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claudio Martella >>>>>> claudio.marte...@gmail.com >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Claudio Martella >>>> claudio.marte...@gmail.com >>>> >>> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> > > > > -- > Kyle Orlando > Computer Engineering Major > University of Maryland > -- Claudio Martella claudio.marte...@gmail.com
