Yes, OOC mechanism in 1.1 may not work in certain scenarios: - If a worker goes into long GC pauses (which is likely since a job that is supposed to use OOC feature usually suffer from high memory pressure), all incoming messages will be buffered before coming to Giraph runtime, and this will cause he job to crash. - The rate at which a worker receives messages may be more than the rate at which a worker spills messages to disk (specially in applications where compute function does a little work, but generates a fair number of messages, such as PageRank). In this case, messages keep accumulating and the worker crashes. - OOC feature is not supported for input superstep for Giraph 1.0 and 1.1. That being said, if graph size is larger than aggregate available memory, you are likely to see the job crashes during input superstep.
OOC feature is under active development at this moment. A new OOC mechanism is recently pushed to the code base (which basically solves the problem for input superstep and OOC graph). However, the new OOC mechanism still have problems for OOC messages. An entirely new approach to OOC is being developed and soon will be available in the code base. Best, Hassan On Thu, Dec 10, 2015 at 8:58 PM, Khaled Ammar <khaled.am...@gmail.com> wrote: > Hi, > > I remember that I used OOC on Giraph 1.0, how ever it is not working on > 1.1. > > Is there a known bug in the OOC feature for recent Giraph versions? > > -- > Thanks, > -Khaled >
