Github user devaraj-kavali commented on a diff in the pull request:
https://github.com/apache/spark/pull/19141#discussion_r138219530
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -565,7 +565,6 @@ private[spark] class Client(
distribute(jarsArchive.toURI.getPath,
resType = LocalResourceType.ARCHIVE,
destName = Some(LOCALIZED_LIB_DIR))
- jarsArchive.delete()
--- End diff --
Thanks @vanzin for the pointer. It was my mistake, I missed the change
reason while looking at the history of the file.
I still see that SPARK-20741 has fixed the issue partially, it leaves
\_\_spark_conf\_\_*.zip file to delete as part of shutdownhook.
I see these approaches to fix it further,
1. Delete \_\_spark_conf\_\_*.zip and \_\_spark_libs\_\_*.zip files after
completing the application similar to cleanupStagingDir.
(Or)
2. Add a configuration whether to delete \_\_spark_conf\_\_*.zip and
\_\_spark_libs\_\_*.zip files after copying to dest dir, so that users can
decide whether to delete these immediately or as part of process exit. In case
of SPARK-20741, this new configuration can be enabled to delete these files
immediately.
@vanzin & @jerryshao Please let me know your thoughts on this or if you
have any other way to do this. Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
