Github user devaraj-kavali commented on a diff in the pull request: https://github.com/apache/spark/pull/19141#discussion_r138219530 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -565,7 +565,6 @@ private[spark] class Client( distribute(jarsArchive.toURI.getPath, resType = LocalResourceType.ARCHIVE, destName = Some(LOCALIZED_LIB_DIR)) - jarsArchive.delete() --- End diff -- Thanks @vanzin for the pointer. It was my mistake, I missed the change reason while looking at the history of the file. I still see that SPARK-20741 has fixed the issue partially, it leaves \_\_spark_conf\_\_*.zip file to delete as part of shutdownhook. I see these approaches to fix it further, 1. Delete \_\_spark_conf\_\_*.zip and \_\_spark_libs\_\_*.zip files after completing the application similar to cleanupStagingDir. (Or) 2. Add a configuration whether to delete \_\_spark_conf\_\_*.zip and \_\_spark_libs\_\_*.zip files after copying to dest dir, so that users can decide whether to delete these immediately or as part of process exit. In case of SPARK-20741, this new configuration can be enabled to delete these files immediately. @vanzin & @jerryshao Please let me know your thoughts on this or if you have any other way to do this. Thanks
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org