Giraph does not offload partitions or messages to HDFS in the out-of-core module. It uses local disk on the computing nodes. By defualt, it uses the tasktracker local directory where for example the distributed cache is stored.
Could you provide the stacktrace Giraph is spitting when failing? On Thu, Sep 12, 2013 at 12:54 AM, Alexander Asplund <alexaspl...@gmail.com>wrote: > Hi, > > I'm still trying to get Giraph to work on a graph that requires more > memory that is available. The problem is that when the Workers try to > offload partitions, the offloading fails. The DiskBackedPartitionStore > fails to create the directory > _bsp/_partitions/job-xxxx/part-vertices-xxx (roughly from recall). > > The input or computation will then continue for a while, which I > believe is because it is still managing to hold everything in memory - > but at some point it reaches the limit where there simply is no more > heap space, and it crashes with OOM. > > Has anybody had this problem with giraph failing to make HDFS directories? > -- Claudio Martella claudio.marte...@gmail.com
