[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13428 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r74680766
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
--- End diff --
Everything else being equal, I'm in favor of leaving this. Maybe it's
redundant but it might prevent bugs in case someone else comes along and tries
to remove the use of absolute paths below (or forgets to use an absolute path
in new code).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r74680747
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
-val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
- "[artifact](-[classifier]).[ext]").mkString(File.separator)
+val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"ivys",
--- End diff --
+1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r74680749
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
-val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
- "[artifact](-[classifier]).[ext]").mkString(File.separator)
+val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"ivys",
+ "ivy.xml").mkString(File.separator)
localIvy.addIvyPattern(localIvyRoot.getAbsolutePath + File.separator +
ivyPattern)
+val artifactPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
+ "[artifact](-[classifier]).[ext]").mkString(File.separator)
+localIvy.addArtifactPattern(localIvyRoot.getAbsolutePath +
File.separator + artifactPattern)
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r73636009
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
-val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
- "[artifact](-[classifier]).[ext]").mkString(File.separator)
+val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"ivys",
--- End diff --
Why not add `localIvyRoot.getAbsolutePath` to this list also and remove the
concatenation in the following statement?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r73636023
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
-val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
- "[artifact](-[classifier]).[ext]").mkString(File.separator)
+val ivyPattern = Seq("[organisation]", "[module]", "[revision]",
"ivys",
+ "ivy.xml").mkString(File.separator)
localIvy.addIvyPattern(localIvyRoot.getAbsolutePath + File.separator +
ivyPattern)
+val artifactPattern = Seq("[organisation]", "[module]", "[revision]",
"[type]s",
+ "[artifact](-[classifier]).[ext]").mkString(File.separator)
+localIvy.addArtifactPattern(localIvyRoot.getAbsolutePath +
File.separator + artifactPattern)
--- End diff --
Same question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r68980204
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
--- End diff --
It's redundant because the filesystem patterns defined below use an
absolute path (with the `localIvyRoot`) and the resolver then ignores the
repository root set here. Only if it determines the patterns are using
relative paths it will then prefix them with the root. It doesn't hurt
anything to leave it in though, if you prefer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13428: [SPARK-12666][CORE] SparkSubmit packages fix for ...
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/13428#discussion_r68973594
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -891,10 +891,12 @@ private[spark] object SparkSubmitUtils {
val localIvy = new FileSystemResolver
val localIvyRoot = new File(ivySettings.getDefaultIvyUserDir, "local")
localIvy.setLocal(true)
-localIvy.setRepository(new FileRepository(localIvyRoot))
--- End diff --
don't we need to keep this for the local IvyCache?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
