xumanbu commented on code in PR #53845:
URL: https://github.com/apache/spark/pull/53845#discussion_r4021992174
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -469,14 +469,20 @@ private[spark] class SparkSubmit extends Logging {
def avoidJarDownload(scheme: String): Boolean =
avoidJarDownloadSchemes.contains("*") ||
avoidJarDownloadSchemes.contains(scheme)
+ val avoidArchiveDownloadSchemes =
sparkConf.get(KUBERNETES_ARCHIVES_AVOID_DOWNLOAD_SCHEMES)
+
+ def avoidArchiveDownload(scheme: String): Boolean =
+ avoidArchiveDownloadSchemes.contains("*") ||
avoidArchiveDownloadSchemes.contains(scheme)
+
val filesLocalFiles = Option(args.files).map {
downloadResourcesToCurrentDirectory(_)
}.orNull
val updatedJars = Option(args.jars).map {
downloadResourcesToCurrentDirectory(_, avoidDownload =
avoidJarDownload)
}.orNull
val archiveLocalFiles = Option(args.archives).map {
- downloadResourcesToCurrentDirectory(_, true)
+ downloadResourcesToCurrentDirectory(_, isArchive = true,
+ avoidDownload = avoidArchiveDownload)
Review Comment:
Thanks for the review - you are right on both counts.
Confirmed on my side as well: the option never actually avoided the driver
download.
`SparkContext` fetches every entry of `spark.archives` when it registers them
(`addArchive` -> `Utils.fetchFile(..., useCache = false)`), so the matched
archives always
reached the driver. The only two things the option could control were
whether the archive is
unpacked into the driver's working directory, and whether `spark.archives`
is rewritten to the
driver-local URI. Binding it to the first one is exactly what broke
SPARK-33748.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
