pengxu created SPARK-6033:
-----------------------------
Summary: the description abou the "spark.worker.cleanup.enabled"
is not matched with the code
Key: SPARK-6033
URL: https://issues.apache.org/jira/browse/SPARK-6033
Project: Spark
Issue Type: Documentation
Components: Documentation
Affects Versions: 1.2.1, 1.2.0
Reporter: pengxu
Some error about the section _Cluster Launch Scripts_ in the
http://spark.apache.org/docs/latest/spark-standalone.html
In the description about the property spark.worker.cleanup.enabled, it states
that *all the directory* under the work dir will be removed whether the
application is running or not.
After checking the implementation in the code level, I found that +only the
stopped application+ dirs would be removed. So the description in the document
is incorrect.
the code implementation in worker.scala
{code: title=WorkDirCleanup}
case WorkDirCleanup =>
// Spin up a separate thread (in a future) to do the dir cleanup; don't
tie up worker actor
val cleanupFuture = concurrent.future {
val appDirs = workDir.listFiles()
if (appDirs == null) {
throw new IOException("ERROR: Failed to list files in " + appDirs)
}
appDirs.filter { dir =>
// the directory is used by an application - check that the
application is not running
// when cleaning up
val appIdFromDir = dir.getName
val isAppStillRunning =
executors.values.map(_.appId).contains(appIdFromDir)
dir.isDirectory && !isAppStillRunning &&
!Utils.doesDirectoryContainAnyNewFiles(dir, APP_DATA_RETENTION_SECS)
}.foreach { dir =>
logInfo(s"Removing directory: ${dir.getPath}")
Utils.deleteRecursively(dir)
}
}
cleanupFuture onFailure {
case e: Throwable =>
logError("App dir cleanup failed: " + e.getMessage, e)
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
