Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/1609#discussion_r15565552
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -947,6 +958,34 @@ private[spark] object Utils extends Logging {
}
false
}
+ /*
+ @volatile private var shutdownStarted = false
+ private[spark] def setShutdownStarted() {
+ shutdownStarted = true
+ }
+
+ def inShutdown(): Boolean = {
+ if (shutdownStarted) return true
+ doShutdownCheck()
+ shutdownStarted
+ }
+
+ private[spark] def doShutdownCheck() {
+ var shutdown = false
+ try {
+ val hook = new Thread {
+ override def run() {}
+ }
+ Runtime.getRuntime.addShutdownHook(hook)
+ Runtime.getRuntime.removeShutdownHook(hook)
+ } catch {
+ case ise: IllegalStateException =>
+ shutdown = true
+ } finally {
+ shutdownStarted = shutdown
+ }
+ }
+ */
--- End diff --
Sure, will move it out when I push next.
It becomes directly relevant to this patch since there are assertions which
check for either file/directory being in expected state or VM is in shutdown
(and so cleanup happened/is happening - which caused file deletions).
For VM shutdown, this is just an optimization - but for shutdown ordered by
driver, inShutdown will return false, but same codepath as shutdown is invoked
by spark (stop on various subsystems) : resulting in exceptions/assertion
failures in threads which are still running.
Unfortunately, this diff interacts badly with local mode - particularly
tests, since it keeps reusing the same VM.
Any ideas on how to 'fix' or resolve this ? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
