RE: How to turn off MetadataCleaner?

2014-05-23 Thread Adrian Mocanu
() ssc.sparkContext.metadataCleaner.cancel() ssc.awaitTermination(500) ssc.stop(true) } -A From: Tathagata Das [mailto:tathagata.das1...@gmail.com] Sent: May-22-14 9:50 PM To: user@spark.apache.org Cc: u...@spark.incubator.apache.org Subject: Re: How to turn off MetadataCleaner? The cleaner should

Re: How to turn off MetadataCleaner?

2014-05-22 Thread Tathagata Das
The cleaner should remain up while the sparkcontext is still active (not stopped). However, here it seems you are stopping the sparkContext (ssc.stop(true)), the cleaner should be stopped. However, there was a bug earlier where some of the cleaners may not have been stopped when the context is