()
ssc.sparkContext.metadataCleaner.cancel()
ssc.awaitTermination(500)
ssc.stop(true)
}
-A
From: Tathagata Das [mailto:tathagata.das1...@gmail.com]
Sent: May-22-14 9:50 PM
To: user@spark.apache.org
Cc: u...@spark.incubator.apache.org
Subject: Re: How to turn off MetadataCleaner?
The cleaner should
Hi
After using sparks TestSuiteBase to run some tests I've noticed that at the
end, after finishing all tests the cleaner is still running and outputs the
following perdiodically:
INFO o.apache.spark.util.MetadataCleaner - Ran metadata cleaner for
SHUFFLE_BLOCK_MANAGER
I use method
The cleaner should remain up while the sparkcontext is still active (not
stopped). However, here it seems you are stopping the sparkContext
(ssc.stop(true)), the cleaner should be stopped. However, there was a bug
earlier where some of the cleaners may not have been stopped when the
context is