xumanbu commented on code in PR #53845:
URL: https://github.com/apache/spark/pull/53845#discussion_r4022038361
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -469,14 +469,20 @@ private[spark] class SparkSubmit extends Logging {
def avoidJarDownload(scheme: String): Boolean =
avoidJarDownloadSchemes.contains("*") ||
avoidJarDownloadSchemes.contains(scheme)
+ val avoidArchiveDownloadSchemes =
sparkConf.get(KUBERNETES_ARCHIVES_AVOID_DOWNLOAD_SCHEMES)
+
+ def avoidArchiveDownload(scheme: String): Boolean =
+ avoidArchiveDownloadSchemes.contains("*") ||
avoidArchiveDownloadSchemes.contains(scheme)
Review Comment:
Good catch. I have fix it :
- The "not unpacked into the working directory" part is fixed by the same
change: archives are
now always unpacked on the driver, whatever the scheme is, `local` and
`file` included.
- For the matching itself, you are right that `*` should not claim to cover
them, so
`executorDirectFetchArchive` now excludes `file` and `local`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
