Re: Scaling Problem

What's your cluster configuration? How you invoke the job? 2013/7/26 jerome richard > Hi, > > I encountered a critical scaling problem using Giraph. I made a very > simple algorithm to test Giraph on large graphs : a connexity test. It > works on relatively large graphs (3 072 441 nodes and 11

Re: Pipes of computation

I think that can be achieved using MasterComputation. Take a look at o.a.g.master.TestSwitchClasses Hope that helps. 2013/7/17 Ahmet Emre Aladağ > Hi, > > Is there a way to pipe the Computation classes? > > Example: Run PageRankComputation and then run NormalizationComputation on > the same gr

Re: Array exception when using out-of-core graph

Sorry guys, it was my bad ... The debug message fooled me ... Really sorry for bothering. 2013/7/4 Avery Ching > Claudio, any thoughts? > > > On 7/3/13 3:52 AM, Han JU wrote: > > Hi, > > I've been testing some algorithm using the out-of-core fea

Array exception when using out-of-core graph

Hi, I've been testing some algorithm using the out-of-core feature, and I have an strange ArrayIndexOutOfBoundsException. In my computation class, the vertex value is a custom writable class which contains a long[]. And during the computation, when the code access this array (say at index 0), the

Aggregate waiting time

Hi, When I examine the logs of each worker, I found that some time is spent on waiting for other tasks: INFO org.apache.giraph.utils.TaskIdsPermitsBarrier: waitForRequiredPermits: Waiting for 7 more tasks to send their aggregator data, task ids: [3, 6, 13, 17, 18, 26, 29] And this seems to me th

Re: The best way to debug

Hi, For my Giraph program, I write tests similar to those in the Giraph project, using the BspCase base class and the InternalVertexRunner. So the job is executed locally in a sequential fashion. When faced with some problems, I usually set break points and do a step by step debug. Have a look at

How to use a custom vertex in 1.1.0 SNAPSHOT

Hi, I changed to version 1.1.0 for the input filtering thing and I noticed the API changes due to GIRAPH-687. Actually this broke my code because, for some reason, I have some instance variables in the vertex class (some flags, lists) apart from the vertex value. When I adapt the code to use the c

Re: What if the resulting graph is larger than the memory?

row > out the few most popular items (if your application allows that). > > Best, > Sebastian > > > On 21.05.2013 12:10, Han JU wrote: > > Hi Sebastian, > > > > It's something like frequent item pairs out of transaction data. > > I need all these pairs

Re: What if the resulting graph is larger than the memory?

sult? > > Best, > Sebastian > > On 21.05.2013 11:52, Han JU wrote: > > Hi Maja, > > > > The input graph of my problem is not big, the calculation result is very > > big. > > In fact what does out-of-core graph mean? Where can I find some examples >

Re: What if the resulting graph is larger than the memory?

you might > want to try out. When that is turned on, during each superstep writeVertex > will be called immediately after compute for that vertex is called. This > means that you can store data you want to write in vertex, write it and > clear the data before going to the next vert

Re: Running Shortest Paths with edge list

maybe change the command: > hadoop jar > target/giraph-0.2-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsVertex -eif org.apache.giraph.io.TextEdgeInputFormat -eip Relationships-Companies-FinancialOrg.tsv -of o

Re: Running Shortest Paths with edge list

Which class? Have you verified every class? 2013/5/18 Rui Sarmento > Hi, > > Continues to throw class not found exception. > > Regards > > -- > Date: Sat, 18 May 2013 19:40:22 +0200 > Subject: Re: Running Shortest Paths with edge list > From: ju.han.fe...@gmail.com >

Re: Running Shortest Paths with edge list

I think the input format class should be org.apache.giraph.io.formats.TextEdgeInputFormat You can verified them by looking at the source code. 2013/5/18 Rui Sarmento > Hi, > > I'm having trouble running the following example with a relatively small > network edge list, can someone please give

What if the resulting graph is larger than the memory?

Hi, It's me again. After a day's work I've coded a Giraph solution for my problem at hand. I gave it a run on a medium dataset and it's notably faster than other approaches. However the goal is to process larger inputs, for example I've a larger dataset that the result graph is about 400GB when r

Re: Questions on input/output format

rtexValueInputFormat. >> You can see an example in TestEdgeInput. >> >> Alessandro >> >> From: Han JU >> Reply-To: "user@giraph.apache.org" >> Date: Wednesday, May 15, 2013 9:00 AM >> To: "user@giraph.apache.org" >> Subje

Re: NullPointerException at the last superstep

Sorry guys.. my mistake.. It was a nasty bug in my code of output format. Sorry for bothering. 2013/5/17 Han JU > Hi, > > I'm developing my first application based on Giraph 1.0.0. > I make it run on our cluster. It's a job with fixed number of supersteps. > Everythin

NullPointerException at the last superstep

Hi, I'm developing my first application based on Giraph 1.0.0. I make it run on our cluster. It's a job with fixed number of supersteps. Everything runs smoothly until the last superstep, this Exception got caught in every worker: java.lang.IllegalStateException: run: Caught an unrecoverable exce

Re: Questions on input/output format

error:" + > error); > return new Text(line); > > I hope I didn't make it more complicated :) > Cheers, > > On Wed, May 15, 2013 at 12:27 PM, Han JU wrote: > >> Hi, >> >> Some questions: >> >> - My input file is a text file with edges:

Questions on input/output format

Hi, Some questions: - My input file is a text file with edges: node1 node2 edgeValue, I figured it out that I should use TextEdgeInputFormat and TextVertexValueInputFormat. But how do these two things fit together? Should I prepare another file that contains only the node informations for Verte

About v1.0

Hi, I'm looking forward to implement a Giraph based solution of my problem at hand at sometime next week. I've read the mails about v1.0 and just want to know when it will come out? Will it have major changes wrt the trunk right now? Thanks! -- *JU Han* Software Engineer Intern @ KXEN Inc. UTC

Re: Beginner's question

It complains about these dependencies: mavanagaiata:0.4.1:branch mavanagaiata:0.4.1:commit maven-assembly-plugin:2.4:single munge-maven-plugin:1.0:munge I use m2eclipse plugin, I ran also mvn eclipse:eclipse then import, same problem. 2013/3/26 André Kelpe > 2013/3/26 Han JU : > &g

Re: Beginner's question

Thanks André and Eli. I'll configure a zookeeper first. By the way, is it normal that there's some error after I imported the project in eclipse? Thanks. 2013/3/26 André Kelpe > 2013/3/25 Eli Reisman > > I have had trouble with the job-local ZK instances myself, > > including using the 1.0.x

Beginner's question

Hi, I'm just starting to try Giraph to do some experiment on graphs. Here's some questions: - I currently have a small cluster on Hadoop 1.0.2 but I don't use any security things, so I compile with -Phadoop_non_secure? - After compilation by "mvn -Phadoop_non_secure clean install" with