persist and unpersist.
unpersist:Mark the RDD as non-persistent, and remove all blocks for it from
memory and disk
2014-03-19 16:40 GMT+08:00 林武康 vboylin1...@gmail.com:
Hi, can any one tell me about the lifecycle of an rdd? I search through
the official website and still can't figure it out.
Related question:
If I keep creating new RDDs and cache()-ing them, does Spark automatically
unpersist the least recently used RDD when it runs out of memory? Or is an
explicit unpersist the only way to get rid of an RDD (barring the PR
Tathagata mentioned)?
Also, does unpersist()-ing an RDD
Yes, Spark automatically removes old RDDs from the cache when you make new
ones. Unpersist forces it to remove them right away. In both cases though, note
that Java doesn’t garbage-collect the objects released until later.
Matei
On Mar 19, 2014, at 7:22 PM, Nicholas Chammas
Okie doke, good to know.
On Wed, Mar 19, 2014 at 7:35 PM, Matei Zaharia matei.zaha...@gmail.comwrote:
Yes, Spark automatically removes old RDDs from the cache when you make new
ones. Unpersist forces it to remove them right away. In both cases though,
note that Java doesn’t garbage-collect
