[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3658 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71710576 Hey @pwendell, I'll try to get to this soon. But I wanted to get your feedback on my idea for fixing the `network/` dependencies thing before I try to implement it. The way I see it, the cleanest way is to do the Guava shading in the earliest artifact possible; that would be `network/common`. So that artifact would have the honor of providing all the relocated Guava classes to everyone. Since `spark-core` depends on it, everything should work out. The only downside I see to that is that `network/common` would now expose `Optional` and friends when it's not really its fault (`spark-core` demands it). What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71762783 [Test build #26199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26199/consoleFull) for PR 3658 at commit [`3c93e42`](https://github.com/apache/spark/commit/3c93e42a5e9474b33aa53f7fd6f22998d44a8c52). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71765771 LGTM pending tests. @vanzin do you have more work you'd like to do on this or are you happy with it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71769526 [Test build #26199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26199/consoleFull) for PR 3658 at commit [`3c93e42`](https://github.com/apache/spark/commit/3c93e42a5e9474b33aa53f7fd6f22998d44a8c52). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71747360 @vanzin yeah this is a fair point, this would mean that network/common would expose the (un-shaded) Guava classes... a bit clunky. If those classes change in the future we could get into trouble. Still, I think this is the best possible solution, provided it's well documented. If we find that Guava changes this in the future we can consider other options. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71773844 No, I'm done with it. Thanks for taking a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71769535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26199/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3658#discussion_r23565016 --- Diff: pom.xml --- @@ -1276,6 +1279,19 @@ includeorg.spark-project.spark:unused/include /includes /artifactSet + relocations +relocation + patterncom.google.common/pattern + shadedPatternorg.spark-project.guava/shadedPattern + excludes +excludecom/google/common/base/Absent*/exclude --- End diff -- mind adding a quick note here to explain that we expose these in our API so we don't relocate them? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3658#discussion_r23564938
--- Diff: core/pom.xml ---
@@ -408,6 +372,20 @@
/execution
/executions
/plugin
+ plugin
+groupIdorg.apache.maven.plugins/groupId
+artifactIdmaven-shade-plugin/artifactId
+configuration
+ shadedArtifactAttachedfalse/shadedArtifactAttached
+ artifactSet
+includes
+ includecom.google.guava:guava/include
+
includeorg.apache.spark:spark-network-common_${scala.binary.version}/include
+
includeorg.apache.spark:spark-network-shuffle_${scala.binary.version}/include
--- End diff --
Is this going to inline other modules (spark-network-common and
spark-network-shuffle) in the published spark-core jar?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3658#discussion_r23566689
--- Diff: core/pom.xml ---
@@ -408,6 +372,20 @@
/execution
/executions
/plugin
+ plugin
+groupIdorg.apache.maven.plugins/groupId
+artifactIdmaven-shade-plugin/artifactId
+configuration
+ shadedArtifactAttachedfalse/shadedArtifactAttached
+ artifactSet
+includes
+ includecom.google.guava:guava/include
+
includeorg.apache.spark:spark-network-common_${scala.binary.version}/include
+
includeorg.apache.spark:spark-network-shuffle_${scala.binary.version}/include
--- End diff --
Yes. From the first comment:
For this reason, also, the network/ dependencies are shaded into the
spark-core artifact too, so that the raw Guava dependency doesn't leak.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71544279 Hey so a couple questions about this. This does make things simpler overall in terms of the build. Some higher level questions: 1. This adds 1,500 class files and about 2 MB or a 30% in file size to the Spark core jar. Does this suggest any limits on how many overall dependencies we can shade in Spark longer term (i.e. in addition to guava)? 2. Inlining two of our own sub-projects into the spark-core jar seems a bit strange. Why not just have the shaded guava in the little assembly jar we create for YARN? Is there a reason we _must_ use the guava dependency that is provided on the cluster? If we did take the current approach, does this mean that Spark core as advertised in maven central won't depend on Spark's network modules... would we still publish the network modules then? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71545834 Does this suggest any limits on how many overall dependencies we can shade in Spark longer term (i.e. in addition to guava)? If we choose relocation as the official way of fixing potential library version conflicts, we'll end up with a growing spark-core. I don't think there's a limit to how many classes you can add (SPARK-1520 notwithstanding), but it doesn't look very pretty, I agree. I did this mainly to fix some issues with the current way Guava is shaded, though, not as a blueprint for how to fix dependency issues going forward. (It also can confuse certain IDEs that automatically add import statements...) Why not just have the shaded guava in the little assembly jar we create for YARN? That's definitely an option. Should be pretty easy to do if that's the preferred way, but I remember that it was a conscious choice to depend on the Yarn-provided Guava. would we still publish the network modules then? Do they provide any public APIs? If they do, then yes. Otherwise, there would be no need to publish them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71547066 [Test build #26116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26116/consoleFull) for PR 3658 at commit [`b3104fc`](https://github.com/apache/spark/commit/b3104fcb12d5d85f7514f575beeb671816b02251). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71550179 Yeah, so I think it would be better to just shade guava in YARN as well. The main dependency-related constraint was that we didn't want Spark's yarn shuffle service to have any external dependencies (i.e. it should work well out of the box with YARN). But having an inlined shaded dependency doesn't matter in this regard. So I don't see any other compelling reason to use YARN's guava. Overall is easier to understand what is going on if we don't have a special case there. And inlining things that we expose as semi-public classes (there are some developer API's in those modules, IIRC), it's a bit off from the normal maven model. So for all these reasons I think it would be good to just shade it in that jar as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71575594 Hey @vanzin mind bringing this up to date now (I merged #3874). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71557798 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26116/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71557787 [Test build #26116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26116/consoleFull) for PR 3658 at commit [`b3104fc`](https://github.com/apache/spark/commit/b3104fcb12d5d85f7514f575beeb671816b02251). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-71236254 Ping. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69804680 [Test build #25477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25477/consoleFull) for PR 3658 at commit [`941848f`](https://github.com/apache/spark/commit/941848f409739331b1d495c4fadb57dc5c373ded). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `val classServer = new HttpServer(conf, outputDir, new SecurityManager(conf), classServerPort, HTTP class server)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69804693 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25477/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69827040 @pwendell would appreciate some input here. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69678894 [Test build #25442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25442/consoleFull) for PR 3658 at commit [`f78c48a`](https://github.com/apache/spark/commit/f78c48a9a52122839f657de19944a81783ff59e5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69684171 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25442/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-69684164 [Test build #25442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25442/consoleFull) for PR 3658 at commit [`f78c48a`](https://github.com/apache/spark/commit/f78c48a9a52122839f657de19944a81783ff59e5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68974879 [Test build #25133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25133/consoleFull) for PR 3658 at commit [`8053dd4`](https://github.com/apache/spark/commit/8053dd4080283385b868c32554d6e103164e0db4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68978591 [Test build #25133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25133/consoleFull) for PR 3658 at commit [`8053dd4`](https://github.com/apache/spark/commit/8053dd4080283385b868c32554d6e103164e0db4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68978595 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25133/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68077871 [Test build #24792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24792/consoleFull) for PR 3658 at commit [`40b8723`](https://github.com/apache/spark/commit/40b87237d8ad573c9279d99251b95fdb2ff4d3be). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68079908 [Test build #24792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24792/consoleFull) for PR 3658 at commit [`40b8723`](https://github.com/apache/spark/commit/40b87237d8ad573c9279d99251b95fdb2ff4d3be). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-68079909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24792/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/3658 [SPARK-4809] Rework Guava library shading. The current way of shading Guava is a little problematic. Code that depends on spark-core does not see the transitive dependency, yet classes in spark-core actually depend on Guava. So it's a little tricky to run unit tests that use spark-core classes, since you need a compatible version of Guava in your dependencies when running the tests. This can become a little tricky, and is kind of a bad user experience. This change modifies the way Guava is shaded so that it's applied uniformly across the Spark build. This means Guava is shaded inside spark-core itself, so that the dependency issues above are solved. Aside from that, all Spark sub-modules have their Guava references relocated, so that they refer to the relocated classes now packaged inside spark-core. Before, this was only done by the time the assembly was built, so project that did not end up inside the assembly (such as streaming backends) could still reference the original location of Guava classes). This relocation does not apply to the sub-modules under network/, though. For those cases, we want to keep the Guava dependency alive, since we want to use the same Guava as the rest of the Yarn NM when deploying the auxiliary shuffle service. For this reason, also, the network/ dependencies are shaded into the spark-core artifact too, so that the raw Guava dependency doesn't leak. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark SPARK-4809 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3658.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3658 commit 4a4ed4202eac66bc288c8fcb2107b0608cc1e32f Author: Marcelo Vanzin van...@cloudera.com Date: 2014-11-21T20:25:15Z [SPARK-4809] Rework Guava library shading. The current way of shading Guava is a little problematic. Code that depends on spark-core does not see the transitive dependency, yet classes in spark-core actually depend on Guava. So it's a little tricky to run unit tests that use spark-core classes, since you need a compatible version of Guava in your dependencies when running the tests. This can become a little tricky, and is kind of a bad user experience. This change modifies the way Guava is shaded so that it's applied uniformly across the Spark build. This means Guava is shaded inside spark-core itself, so that the dependency issues above are solved. Aside from that, all Spark sub-modules have their Guava references relocated, so that they refer to the relocated classes now packaged inside spark-core. Before, this was only done by the time the assembly was built, so project that did not end up inside the assembly (such as streaming backends) could still reference the original location of Guava classes). This relocation does not apply to the sub-modules under network/, though. For those cases, we want to keep the Guava dependency alive, since we want to use the same Guava as the rest of the Yarn NM when deploying the auxiliary shuffle service. For this reason, also, the network/ dependencies are shaded into the spark-core artifact too, so that the raw Guava dependency doesn't leak. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66393496 I ran the maven build + unit tests, and also audited the jar files in the build using the tool I wrote: https://gist.github.com/vanzin/bd9057fadf4a296220b7 I'll run some actual jobs tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66393963 [Test build #24285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24285/consoleFull) for PR 3658 at commit [`4a4ed42`](https://github.com/apache/spark/commit/4a4ed4202eac66bc288c8fcb2107b0608cc1e32f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66399893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24285/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66399889 [Test build #24285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24285/consoleFull) for PR 3658 at commit [`4a4ed42`](https://github.com/apache/spark/commit/4a4ed4202eac66bc288c8fcb2107b0608cc1e32f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4809] Rework Guava library shading.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3658#issuecomment-66404298 BTW, credit where credit is due, I got this idea from @arahuja --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
