Repository: spark Updated Branches: refs/heads/branch-2.1 c3a986b19 -> f5ef07625
[SPARK-20756][YARN] yarn-shuffle jar references unshaded guava and contains scala classes ## What changes were proposed in this pull request? This change ensures that all references to guava from within the yarn shuffle jar pointed to the shaded guava class already provided in the jar. Also, it explicitly excludes scala classes from being added to the jar. ## How was this patch tested? Ran unit tests on the module and they passed. javap now returns the expected result - reference to the shaded guava under `org/spark_project` (previously this was referring to `com.google...` ``` javap -cp common/network-yarn/target/scala-2.11/spark-2.3.0-SNAPSHOT-yarn-shuffle.jar -c org/apache/spark/network/yarn/YarnShuffleService | grep Lists 57: invokestatic #138 // Method org/spark_project/guava/collect/Lists.newArrayList:()Ljava/util/ArrayList; ``` Guava is still shaded in the jar: ``` jar -tf common/network-yarn/target/scala-2.11/spark-2.3.0-SNAPSHOT-yarn-shuffle.jar | grep guava | head META-INF/maven/com.google.guava/ META-INF/maven/com.google.guava/guava/ META-INF/maven/com.google.guava/guava/pom.properties META-INF/maven/com.google.guava/guava/pom.xml org/spark_project/guava/ org/spark_project/guava/annotations/ org/spark_project/guava/annotations/Beta.class org/spark_project/guava/annotations/GwtCompatible.class org/spark_project/guava/annotations/GwtIncompatible.class org/spark_project/guava/annotations/VisibleForTesting.class ``` (not sure if the above META-INF/* is a problem or not) I took this jar, deployed it on a yarn cluster with shuffle service enabled, and made sure the YARN node managers came up. An application with a shuffle was run and it succeeded. Author: Mark Grover <m...@apache.org> Closes #17990 from markgrover/spark-20756. (cherry picked from commit 36309110046a89d749a7c9746eaa16997de26922) Signed-off-by: Marcelo Vanzin <van...@cloudera.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f5ef0762 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f5ef0762 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f5ef0762 Branch: refs/heads/branch-2.1 Commit: f5ef0762518892986aebcabbcc6c269273d2e1be Parents: c3a986b Author: Mark Grover <m...@apache.org> Authored: Mon May 22 10:10:41 2017 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Mon May 22 10:11:04 2017 -0700 ---------------------------------------------------------------------- common/network-yarn/pom.xml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/f5ef0762/common/network-yarn/pom.xml ---------------------------------------------------------------------- diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml index 7a57e89..fb6c241 100644 --- a/common/network-yarn/pom.xml +++ b/common/network-yarn/pom.xml @@ -87,6 +87,9 @@ <includes> <include>*:*</include> </includes> + <excludes> + <exclude>org.scala-lang:scala-library</exclude> + </excludes> </artifactSet> <filters> <filter> @@ -98,7 +101,7 @@ </excludes> </filter> </filters> - <relocations> + <relocations combine.children="append"> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>${spark.shade.packageName}.com.fasterxml.jackson</shadedPattern> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org