GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/3658
[SPARK-4809] Rework Guava library shading. The current way of shading Guava is a little problematic. Code that depends on "spark-core" does not see the transitive dependency, yet classes in "spark-core" actually depend on Guava. So it's a little tricky to run unit tests that use spark-core classes, since you need a compatible version of Guava in your dependencies when running the tests. This can become a little tricky, and is kind of a bad user experience. This change modifies the way Guava is shaded so that it's applied uniformly across the Spark build. This means Guava is shaded inside spark-core itself, so that the dependency issues above are solved. Aside from that, all Spark sub-modules have their Guava references relocated, so that they refer to the relocated classes now packaged inside spark-core. Before, this was only done by the time the assembly was built, so project that did not end up inside the assembly (such as streaming backends) could still reference the original location of Guava classes). This relocation does not apply to the sub-modules under network/, though. For those cases, we want to keep the Guava dependency alive, since we want to use the same Guava as the rest of the Yarn NM when deploying the auxiliary shuffle service. For this reason, also, the network/ dependencies are shaded into the spark-core artifact too, so that the raw Guava dependency doesn't leak. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark SPARK-4809 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3658.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3658 ---- commit 4a4ed4202eac66bc288c8fcb2107b0608cc1e32f Author: Marcelo Vanzin <van...@cloudera.com> Date: 2014-11-21T20:25:15Z [SPARK-4809] Rework Guava library shading. The current way of shading Guava is a little problematic. Code that depends on "spark-core" does not see the transitive dependency, yet classes in "spark-core" actually depend on Guava. So it's a little tricky to run unit tests that use spark-core classes, since you need a compatible version of Guava in your dependencies when running the tests. This can become a little tricky, and is kind of a bad user experience. This change modifies the way Guava is shaded so that it's applied uniformly across the Spark build. This means Guava is shaded inside spark-core itself, so that the dependency issues above are solved. Aside from that, all Spark sub-modules have their Guava references relocated, so that they refer to the relocated classes now packaged inside spark-core. Before, this was only done by the time the assembly was built, so project that did not end up inside the assembly (such as streaming backends) could still reference the original location of Guava classes). This relocation does not apply to the sub-modules under network/, though. For those cases, we want to keep the Guava dependency alive, since we want to use the same Guava as the rest of the Yarn NM when deploying the auxiliary shuffle service. For this reason, also, the network/ dependencies are shaded into the spark-core artifact too, so that the raw Guava dependency doesn't leak. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org