The ContextCleaner uncaches RDDs that have gone out of scope on the driver. So it's possible that the given RDD is no longer reachable in your program's control flow, or else it'd be a bug in the ContextCleaner.
On Wed, Dec 10, 2014 at 5:34 PM, ankits <ankitso...@gmail.com> wrote: > I'm using spark 1.1.0 and am seeing persisted RDDs being cleaned up too > fast. > How can i inspect the size of RDD in memory and get more information about > why it was cleaned up. There should be more than enough memory available on > the cluster to store them, and by default, the spark.cleaner.ttl is > infinite, so I want more information about why this is happening and how to > prevent it. > > Spark just logs this when removing RDDs: > > [2014-12-11 01:19:34,006] INFO spark.storage.BlockManager [] [] - Removing > RDD 33 > [2014-12-11 01:19:34,010] INFO pache.spark.ContextCleaner [] > [akka://JobServer/user/context-supervisor/job-context1] - Cleaned RDD 33 > [2014-12-11 01:19:34,012] INFO spark.storage.BlockManager [] [] - Removing > RDD 33 > [2014-12-11 01:19:34,016] INFO pache.spark.ContextCleaner [] > [akka://JobServer/user/context-supervisor/job-context1] - Cleaned RDD 33 > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/RDDs-being-cleaned-too-fast-tp20613.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
