Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31491951
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -91,51 +91,54 @@ private[spark] class Client(
* available in the alpha API.
*/
def submitApplication(): ApplicationId = {
- var appId: ApplicationId = null
+ // Before we submit current application, we cleanup staging director
as some old appStagingDir
+ // can not be deleted when those old jobs are failed or killed and so
on, please see SPARK-7705
+ // and SPARK-7503 for details.
+ cleanupStagingDir()
+
+ // Setup the credentials before doing anything else, so we have don't
have issues at any point.
+ setupCredentials()
+ yarnClient.init(yarnConf)
+ yarnClient.start()
+
+ logInfo("Requesting a new application from cluster with %d
NodeManagers"
+ .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+
+ // Get a new application from our RM
+ val newApp = yarnClient.createApplication()
+ val newAppResponse = newApp.getNewApplicationResponse()
+ val appId = newAppResponse.getApplicationId()
+
+ // Verify whether the cluster has enough resources for our AM
+ verifyClusterResources(newAppResponse)
+
+ // Set up the appropriate contexts to launch our AM
+ val containerContext = createContainerLaunchContext(newAppResponse)
+ val appContext = createApplicationSubmissionContext(newApp,
containerContext)
+
+ // Finally, submit and monitor the application
+ logInfo(s"Submitting application ${appId.getId} to ResourceManager")
+ yarnClient.submitApplication(appContext)
+ appId
+ }
+
+ /**
+ * Cleanup all subdirectory of SPARK_STAGING directory.
+ */
+ private def cleanupStagingDir(): Unit = {
+ val stagingDirPath = new Path(SPARK_STAGING)
--- End diff --
No no no no no no no. You're now adding a horrible bug here. Because if you
launch two different apps, one will delete the other's files.
Look at `getAppStagingDir` and how it includes the app id in the path to
avoid that problem.
When I mentioned earlier to try to do cleanup in the launcher, I never
meant to do it like this. I meant to do the cleanup *after* the app finishes,
in case the staging directory is still around.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
