Re: Spark handling spill overs

Spill-overs are a common issue for in-memory computing systems, after all memory is limited. In Spark where RDDs are immutable, if an RDD got created with its size > 1/2 node's RAM then a transformation and generation of the consequent RDD' can potentially fill all the node's memory that can

Re: Spark handling spill overs

Hi, Which version of Spark you use? The recent one cannot handle this kind of spilling, see: http://spark.apache.org/docs/latest/tuning.html#memory-management-overview. // maropu On Fri, May 13, 2016 at 8:07 AM, Ashok Kumar wrote: > Hi, > > How one can avoid

Spark handling spill overs

Hi, How one can avoid having Spark spill over after filling the node's memory. Thanks