As David mentions, even with OOC, the objects are still created (and yes,
often soon destroyed after spilled to disk) putting pressure on the GC.
Moreover, with the increase in size of the graph, the number of in-memory
vertices is not the only increasing chunk of memory, as there are other
memory
Thanks, disabling GC overhead limit did the trick!
I did however run into another issue - the computation ends up
stalling when it tries to write partitions to disk. All the workers
keep sending out messages about DiskBackedPartitionStore failed to
create directory
Correction: the computation does not actually stall - it does
complains a bit that the directories cannot be created and then
eventually moves to the next superstep. I guess this means I'm
actually fitting all the data in memory?
On 9/10/13, Alexander Asplund alexaspl...@gmail.com wrote:
Thanks,
Thanks for the reply.
I tried setting giraph.maxPartitionsInMemory to 1, but I'm still
getting OOM: GC limit exceeded.
Are there any particular cases the OOC will not be able to handle, or
is it supposed to work in all cases? If the latter, it might be that I
have made some configuration error.
did you extend the heap available to the mapper tasks? e.g. through
mapred.child.java.opts.
On Tue, Sep 10, 2013 at 12:50 AM, Alexander Asplund
alexaspl...@gmail.comwrote:
Thanks for the reply.
I tried setting giraph.maxPartitionsInMemory to 1, but I'm still
getting OOM: GC limit exceeded.
Really appreciate the swift responses! Thanks again.
I have not both increased mapper tasks and decreased max number of
partitions at the same time. I first did tests with increased Mapper
heap available, but reset the setting after it apparently caused
other, large volume, non-Giraph jobs to
A small note: I'm not seeing any partitions directory being formed
under _bsp, which is where I have understood that they should be
appearing.
On 9/10/13, Alexander Asplund alexaspl...@gmail.com wrote:
Really appreciate the swift responses! Thanks again.
I have not both increased mapper tasks
Alexander:
You might try turning off the GC Overhead limit
(-XX:-UseGCOverheadLimit)
Also you could turn on verbose GC logging (-verbose:gc
-Xloggc:/tmp/@taskid@.gc)
to see what is happening.
Because the OOC still has to create and destroy objects I suspect that
the heap is just
getting
OOC is used also at input superstep. try to decrease the number of
partitions kept in memory.
On Sat, Sep 7, 2013 at 1:37 AM, Alexander Asplund alexaspl...@gmail.comwrote:
Hi,
I'm trying to process a graph that is about 3 times the size of
available memory. On the other hand, there is
