Re: Reading Back a Cached RDD

Nick, what is your use-case? On Thu, Mar 24, 2016 at 11:55 PM, Marco Colombo wrote: > You can persist off-heap, for example with tachyon, now called Alluxio. > Take a look at off heap peristance > > Regards > > > Il giovedì 24 marzo 2016, Holden Karau

Re: Reading Back a Cached RDD

You can persist off-heap, for example with tachyon, now called Alluxio. Take a look at off heap peristance Regards Il giovedì 24 marzo 2016, Holden Karau ha scritto: > Even checkpoint() is maybe not exactly what you want, since if reference > tracking is turned on it will

Re: Reading Back a Cached RDD

Even checkpoint() is maybe not exactly what you want, since if reference tracking is turned on it will get cleaned up once the original RDD is out of scope and GC is triggered. If you want to share persisted RDDs right now one way to do this is sharing the same spark context (using something like

Re: Reading Back a Cached RDD

Isn’t persist() only for reusing an RDD within an active application? Maybe checkpoint() is what you’re looking for instead? ​ On Thu, Mar 24, 2016 at 2:02 PM Afshartous, Nick wrote: > > Hi, > > > After calling RDD.persist(), is then possible to come back later and >

Reading Back a Cached RDD

Hi, After calling RDD.persist(), is then possible to come back later and access the persisted RDD. Let's say for instance coming back and starting a new Spark shell session. How would one access the persisted RDD in the new shell session ? Thanks, -- Nick