@Claudio: Sorry I wasn't clear, I meant it is inherent that every step in BFS will cost you a superstep and a complete iteration over the graph. In other programming paradigms (like a C program with a bunch of pointers) this traversal could be implemented more efficiently.
On Fri, Feb 8, 2013 at 2:35 AM, pradeep kumar <pradeep0...@gmail.com> wrote: > Hi Dionysis, > > i was getting those error's because of the configuration issues (even > after reading all the input values correctly it was not able to start > compute for superstep 0).. it was because i was running program in wrong > way.. > > (you can check till where your program is running by putting some print > st's and checking o/p in syslog) > > following cmd worked for me > > hadoop jar giraph-0.2-SNAPSHOT-for-hadoop-0.20.2-jar-with-dependencies.jar > org.apache.giraph.examples.bc_random > org.apache.giraph.examples.bc_randomVertex -_GMInputFormat ADJ > org.apache.giraph.examples.bc_randomVertexInputFormat -_GMOutputFormat ADJ > org.apache.giraph.examples.bc_randomVertexOutputFormat --K 5 -i inputbet -o > out -w 1 > > > @Claudio : Sorry actually my prog was exceeding counter limit which got > resolved by increasing the the counter limit in mapred (gave 1024 i kw its > never gona hit even close to that :-)) as you suggested, > > But its was just for small sample input (more of like linear graph), i am > worried about actual graph which is huge and i might not even know the longest > shortest path in graph (randomly changing) so.. > > was just asking is there a way i can cancel checking counter limit check.. > > > > On Fri, Feb 8, 2013 at 2:51 AM, Claudio Martella < > claudio.marte...@gmail.com> wrote: > >> Sorry pradeep, I'm not sure I understand your problem. What about the 361 >> supersteps and the counters limit change? >> Did you solve it? >> >> >> On Thu, Feb 7, 2013 at 5:43 PM, pradeep kumar <pradeep0...@gmail.com>wrote: >> >>> Hi Claudio >>> actually i did increased limit on my laptop to 1024 (pseudo mode) it >>> took 361 supersteps for 15 nodes and 18 edges (depth 10+), we will soon be >>> testing this on cluster of 5 or 6 hope everything goes well. Is there a way >>> i can disable this check..? i remember someone posting this earlier..!! but >>> not sure how..;-( >>> >>> @Jan, yes, actually even i want to see results,..:-) >>> >>> PS thanks for all help.. >>> >>> >>> >>> >>> On Thu, Feb 7, 2013 at 9:47 PM, Jan van der Lugt <janl...@gmail.com>wrote: >>> >>>> I never actually ran bc_random, so I don't know how many supersteps it >>>> needs for different input sizes. If you could try this out, it would be >>>> worthwhile to know. Especially the relation between input size / number of >>>> supersteps. It's a limitation of the Pregel paradigm that you need quite a >>>> few superstep to do BFS, not a lot we can do about this... >>>> >>>> >>>> On Thu, Feb 7, 2013 at 3:47 PM, Claudio Martella < >>>> claudio.marte...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> you have to increase the counters limit in your cluster. You can >>>>> increase the limit by setting the property mapreduce.job.counters.limit in >>>>> your mapred-site.xml accordingly. >>>>> >>>>> >>>>> On Thu, Feb 7, 2013 at 4:34 PM, pradeep kumar >>>>> <pradeep0...@gmail.com>wrote: >>>>> >>>>>> Hi Jan, >>>>>> >>>>>> well actually i just corrected my mistake.. i was running it wrong >>>>>> way, thanks.. >>>>>> just 1 more question on map-reduce counters >>>>>> >>>>>> giraph by default limits the superstep counters to 120 >>>>>> but in bc_random master superstep itself exceeds this, so i had to >>>>>> increase the limit..this was just for 20 nodes sample i/p i was trying >>>>>> >>>>>> my real data will be close to million nodes, so how can i manage for >>>>>> such large input.. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Feb 7, 2013 at 4:02 PM, pradeep kumar >>>>>> <pradeep0...@gmail.com>wrote: >>>>>> >>>>>>> Hi lugt, >>>>>>> >>>>>>> Thanks for all your help.. program compiled successfully into jar.. >>>>>>> but actually i am not able to run programs.. >>>>>>> i tried following cmd to run >>>>>>> >>>>>>> 1) using GiraphRunner >>>>>>> hadoop jar >>>>>>> giraph-0.2-SNAPSHOT-for-hadoop-0.20.2-jar-with-dependencies.jar >>>>>>> org.apache.giraph.GiraphRunner >>>>>>> org.apache.giraph.examples.bc_randomVertex >>>>>>> -vif org.apache.giraph.examples.bc_randomVertexInputFormat -vip >>>>>>> inputbet >>>>>>> -of org.apache.giraph.examples.bc_randomVertexOutputFormat -op >>>>>>> outputbet >>>>>>> -w 1 >>>>>>> >>>>>>> it does maps every thing successfully but then gives following error >>>>>>> in log >>>>>>> >>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception >>>>>>> waitFor: ExecutionException occurred while waiting for >>>>>>> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@40fd6cd8 >>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102) >>>>>>> at >>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) >>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) >>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:266) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>>> >>>>>>> followed by some other errors >>>>>>> >>>>>>> 2) bc_random >>>>>>> >>>>>>> i actually couldnt understand all arg types here >>>>>>> >>>>>>> used : >>>>>>> >>>>>>> hadoop jar >>>>>>> giraph-0.2-SNAPSHOT-for-hadoop-0.20.2-jar-with-dependencies.jar >>>>>>> org.apache.giraph.examples.bc_random >>>>>>> org.apache.giraph.examples.bc_randomVertex --GMInputFormat ADJ >>>>>>> org.apache.giraph.examples.bc_randomVertexInputFormat -i inputbet >>>>>>> --GMOutputFormat ADJ >>>>>>> org.apache.giraph.examples.bc_randomVertexOutputFormat -o outputbet -w 1 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> but gives error for k procedure arg (not sure what to give as value >>>>>>> here, even about about ip format i mentioned above ) >>>>>>> >>>>>>> >>>>>>> I have uploaded sample input in hdfs in following format >>>>>>> >>>>>>> 10 4500 9 800 11 1000 12 1000 >>>>>>> 11 5500 12 1100 11 800 >>>>>>> >>>>>>> nid val <eid val> >>>>>>> >>>>>>> did i made a mistake anywhere..? >>>>>>> >>>>>>> any suggestion will be a great help..!! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 6, 2013 at 5:03 PM, Jan van der Lugt >>>>>>> <janl...@gmail.com>wrote: >>>>>>> >>>>>>>> Hi Pradeep, >>>>>>>> >>>>>>>> You can check out the Green-Marl compiler from my Github ( >>>>>>>> https://github.com/janlugt/Green-Marl), that one should be >>>>>>>> compatible with the latest version of Giraph. Please let me know if >>>>>>>> anything doesn't work for you. The changes will probably also be >>>>>>>> accepted >>>>>>>> upstream later this day. >>>>>>>> >>>>>>>> - Jan >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 6, 2013 at 9:48 AM, Jan van der Lugt <janl...@gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> It should work with version 0.2, but let me check. Since the API >>>>>>>>> is changing all the time at the moment, it's very different to track >>>>>>>>> all >>>>>>>>> those changes. Probably they are numerous instances of the same error. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 6, 2013 at 9:26 AM, pradeep kumar < >>>>>>>>> pradeep0...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Lugt, >>>>>>>>>> >>>>>>>>>> Thanks for suggestion, i just tried with Green-Marl's bc_random, >>>>>>>>>> But because of so many changes in Giraph codebase, imports and >>>>>>>>>> many other functions in bc_random >>>>>>>>>> generate lots of errors almost 100+ errors.. I tried with Giraph >>>>>>>>>> 1.0 and current version of 2.0, >>>>>>>>>> most of errors i have resolved but still there are many errors.. >>>>>>>>>> Any suggestion on this..? >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pradeep Kumar >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pradeep Kumar >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claudio Martella >>>>> claudio.marte...@gmail.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Pradeep Kumar >>> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> > > > > -- > Pradeep Kumar >
