Re: Out of core execution has no effect on GC crash

As David mentions, even with OOC, the objects are still created (and yes, often soon destroyed after spilled to disk) putting pressure on the GC. Moreover, with the increase in size of the graph, the number of in-memory vertices is not the only increasing chunk of memory, as there are other memory

Re: Counter limit

Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years

Re: Counter limit

one the command line, you can use the -D option after the GiraphRunner class before the GiraphRunner specific parameters, e.g. -D giraph. useSuperstepCounters=false On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause m...@ckrause.org wrote: Thanks a lot. One last question: where do I set options

Re: Finding missing links in a lineage graph..

Hi Sashant, you'll have to write your own algorithm that acts depending on the labels along the edges. On Tue, Sep 10, 2013 at 9:46 AM, Sushanta Pradhan sushanta.prad...@talentica.com wrote: Hi, I am trying to create a lineage graph from an incomplete data i.e. few relationships are

Re: Counter limit

I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my MasterCompute class? Cheers, Christian 2013/9/10 Claudio Martella claudio.marte...@gmail.com one the command line, you can use the -D option after the GiraphRunner

Re: Counter limit

you can set it in your giraph-site.xml, but it should work on the command line. On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause m...@ckrause.org wrote: I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my

Re: Out of core execution has no effect on GC crash

Thanks, disabling GC overhead limit did the trick! I did however run into another issue - the computation ends up stalling when it tries to write partitions to disk. All the workers keep sending out messages about DiskBackedPartitionStore failed to create directory

Re: Out of core execution has no effect on GC crash

Correction: the computation does not actually stall - it does complains a bit that the directories cannot be created and then eventually moves to the next superstep. I guess this means I'm actually fitting all the data in memory? On 9/10/13, Alexander Asplund alexaspl...@gmail.com wrote: Thanks,