Job complete but no results

Hi All, I'm trying to run ConnectedComponents example but the job is killed for some reason I can't discover, can you please help? Many Thanks in advance. > hadoop jar > target/giraph-0.2-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.g

Re: What if the resulting graph is larger than the memory?

Sounds like you could make use of HBase to store your results. The vertices could simply be row keys into HBase... On Tue, May 21, 2013 at 2:52 AM, Han JU wrote: > Hi Maja, > > The input graph of my problem is not big, the calculation result is very > big. > In fact what does out-of-core graph

Re: What if the resulting graph is larger than the memory?

Let me understand. You said that your graph is about 400GB, and that you don't have 400GB of main memory in your cluster (assuming for now that 400GB of edge-based input format would actually result in that amount of memory used on the heap, which is not the case). If THIS is your problem, and not

Re: What if the resulting graph is larger than the memory?

It simply means that not all partitions of the graph are in-memory all the time. If you don't have enugh memory, some of them might get spilled to disk. On 21.05.2013 14:16, Han JU wrote: > Thanks, that's a good point. > But for the moment I just want to try out different solutions on hadoop and >

Re: What if the resulting graph is larger than the memory?

Thanks, that's a good point. But for the moment I just want to try out different solutions on hadoop and have a comparison of them. So I'd like to see how they perform under general conditions. Do you happen to know what out-of-core graph means? Thanks. 2013/5/21 Sebastian Schelter > Ah, I se

Re: Superstep count

OK, I found it. Sample code for archive purposes (here it's 3 supersteps, if config doesnt have the key): In the compute method: SUPERSTEP_COUNT = "giraph.pageRank.superstepCount"; if (superStep < getConf().getInt(SUPERSTEP_COUNT, 3)) { sendMessageToAllEdges(

Re: What if the resulting graph is larger than the memory?

Ah, I see. I have worked on similar things in recommender systems. Here the problem is generally that you get a result quadratic to the number of interactions per item. If you have some topsellers in your data, those might make up for the large result. It helps very much to throw out the few most p

Re: What if the resulting graph is larger than the memory?

Hi Sebastian, It's something like frequent item pairs out of transaction data. I need all these pairs with somehow a low support (say 2), so the result could be very big. 2013/5/21 Sebastian Schelter > Hello Han, > > out of curiosity, what do you compute that has such a big result? > > Best,

Re: What if the resulting graph is larger than the memory?

Hello Han, out of curiosity, what do you compute that has such a big result? Best, Sebastian On 21.05.2013 11:52, Han JU wrote: > Hi Maja, > > The input graph of my problem is not big, the calculation result is very > big. > In fact what does out-of-core graph mean? Where can I find some exampl

Re: What if the resulting graph is larger than the memory?

Hi Maja, The input graph of my problem is not big, the calculation result is very big. In fact what does out-of-core graph mean? Where can I find some examples of this and for output during computation? Thanks. 2013/5/17 Maja Kabiljo > Hi JU, > > One thing you can try is to use out-of-core