Yes for the zookeeper problem I passed a seperate jar through -libjars command. If I use additional jars zookeeper fails. On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.marte...@gmail.com> wrote:
> there must be a misunderstanding. i was referring to the zookeeper problem. > > > On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium > <vivek.semb...@gmail.com>wrote: > >> No. I added my files(it was just a copy of one of the example program to >> a different package) to the jar files of giraph. But it was still giving me >> classNotFoundException. Can you give me some simple example program with >> instructions on how to deploy it. So I can start playing with giraph and >> make changes to the program and learn, then start working on my project in >> giraph. >> I will be very thankful if you can help me with this. >> >> Thanking you >> -Vivek Sembium >> >> >> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella < >> claudio.marte...@gmail.com> wrote: >> >>> but you were still using an additional jar added through -libjars, right? >>> >>> >>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium >>> <vivek.semb...@gmail.com>wrote: >>> >>>> @Claudio Martella Your solution didnt work either. I basically tried >>>> copying the pageRankBenchmark to my own package, renamed the package. It >>>> compiles fine with giraph. But I couldnt run it even if I add those files >>>> to giraph jar before deployment. Help? >>>> >>>> >>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella < >>>> claudio.marte...@gmail.com> wrote: >>>> >>>>> you have this problem when you use two jars (one with giraph and one >>>>> with your classes) instead of a single fat-jar, correct? I tracked the >>>>> same >>>>> problem a few weeks ago, basically zookeeper is run passing the wrong jar. >>>>> >>>>> >>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium < >>>>> vivek.semb...@gmail.com> wrote: >>>>> >>>>>> Thank you for your suggestion. It worked. Its not giving class not >>>>>> found exception. But its giving me a new error >>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found that >>>>>> its unable to connect to zookeeper service. >>>>>> >>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable >>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries! >>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) >>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>> at >>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>>>> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers: >>>>>> Failed to connect in 10 tries! >>>>>> at >>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727) >>>>>> at >>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371) >>>>>> at >>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204) >>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59) >>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89) >>>>>> ... 7 more >>>>>> >>>>>> Immediately I ran page rank benchmark and it executed successfully >>>>>> both from giraph in lib directory and also from giraphs own directory. >>>>>> >>>>>> Can you give me a very simple java program(finding maximum in a graph >>>>>> or simple page rank program) in giraph along with its jar file and input >>>>>> files which I can place in my lib directory of hadoop and test if its >>>>>> working. And also the command to execute it. This should be added in the >>>>>> documentation as new comers can quickly setup giraph and concentrate on >>>>>> their project. >>>>>> >>>>>> >>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ < >>>>>> emre.ala...@agmlab.com> wrote: >>>>>> >>>>>>> It isn't asking for edge input. It says make sure you don't need it. >>>>>>> A warning for the case you may have forgotten to give edge input when >>>>>>> you >>>>>>> really needed. >>>>>>> >>>>>>> The cause of your error is what I'm wondering nowadays. I'm having a >>>>>>> similar problem. Currently I'm using a workaround: put all the jars >>>>>>> (giraph-core and my module giraph-nutch) in the lib folder of hadoop. >>>>>>> Then >>>>>>> it works. But there should be a clean way of doing this. >>>>>>> >>>>>>> I should be able to say hadoop jar fat.jar ... >>>>>>> >>>>>>> Any help appreciated. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> *Kimden: *"Vivek Sembium" <vivek.semb...@gmail.com> >>>>>>> *Kime: *user@giraph.apache.org >>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49 >>>>>>> *Konu: *Re: Help needed for Running my own java programs in Giraph >>>>>>> >>>>>>> >>>>>>> I tried with and without exporting hadoop classpath. I get the same >>>>>>> error. >>>>>>> >>>>>>> Here's the command that I tried >>>>>>> hadoop jar >>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar >>>>>>> org.apache.giraph.GiraphRunner -libjars >>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar >>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute -vif >>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>>>> -vip /user/vivek/tiny_graph.txt -w 3 >>>>>>> >>>>>>> >>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input >>>>>>> format specified. Ensure your InputFormat does not require one. >>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format >>>>>>> specified. Ensure your OutputFormat does not require one. >>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>>> org/apache/giraph/master/DefaultMasterCompute >>>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:634) >>>>>>> at >>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>>> at >>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277) >>>>>>> at java.net.URLClassLoader.access$000(URLClassLoader.java:73) >>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:212) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>> at >>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:314) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>> at java.lang.Class.forName(Class.java:186) >>>>>>> at >>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425) >>>>>>> at >>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408) >>>>>>> at >>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200) >>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74) >>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>>>> Method) >>>>>>> at >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>>>>> Caused by: java.lang.ClassNotFoundException: >>>>>>> org.apache.giraph.master.DefaultMasterCompute >>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321) >>>>>>> at >>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >>>>>>> ... 26 more >>>>>>> >>>>>>> >>>>>>> I gave the graph input as JSON file. Why does it ask for edge input >>>>>>> file again? >>>>>>> >>>>>>> >>>>>>> Why isnt this working? >>>>>>> >>>>>>> >>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando < >>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Vivek, >>>>>>>> >>>>>>>> Sorry for the hasty response, I assumed the quick start guide >>>>>>>> covered that (I guess it doesn't). What you need to do first is put >>>>>>>> all of >>>>>>>> your compiled code into a jar file. Then, you'll be utilizing the >>>>>>>> hadoop >>>>>>>> option -libjars. You also need to make sure that you set the hadoop >>>>>>>> classpath to include the giraph jar you are using as well as your own >>>>>>>> jar. >>>>>>>> I actually asked a similar question not too long ago; prior to this, I >>>>>>>> was >>>>>>>> just combining everything into one jar. >>>>>>>> >>>>>>>> >>>>>>>> Your code should pretty much follow this format, which is copied >>>>>>>> and paster from a previous response by Claudio: >>>>>>>> >>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar" >>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D option >>>>>>>> ]* -libjars your.jar your.package.ComputationClass [ GiraphRunner >>>>>>>> option e.g. -vip -vif etc. ]* >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium < >>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thank You for your response. I could run all the example programs. >>>>>>>>> I want to place my development code in a different directory other >>>>>>>>> than the >>>>>>>>> giraph directory. I copied the SimplePageRankComputation.java to a >>>>>>>>> different directory, compiled it(with a different package name). But I >>>>>>>>> couldnt replace the >>>>>>>>> "org.apache.giraph.examples.SimpleShortestPathsComputation" >>>>>>>>> with my class file. It gives me a ClassNotFoundException. Please >>>>>>>>> Guide me >>>>>>>>> so that I can run any class file with all required methods (GIven the >>>>>>>>> location of that file). >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando < >>>>>>>>> kyle.r.orla...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> A "Quick Start" guide was recently added to the Giraph website. >>>>>>>>>> Now learning Giraph is easier than ever! Here: >>>>>>>>>> http://giraph.apache.org/quick_start.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium < >>>>>>>>>> vivek.semb...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I am a new user in giraph. I could run all example programs in >>>>>>>>>>> Giraph. Can someone please guide me on how to run my own java >>>>>>>>>>> program in >>>>>>>>>>> giraph (Assuming I have my input file in jason format in HDFS). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Kyle Orlando >>>>>>>>>> Computer Engineering Major >>>>>>>>>> University of Maryland >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Kyle Orlando >>>>>>>> Computer Engineering Major >>>>>>>> University of Maryland >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Claudio Martella >>>>> claudio.marte...@gmail.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Claudio Martella >>> claudio.marte...@gmail.com >>> >> >> > > > -- > Claudio Martella > claudio.marte...@gmail.com >
