[ https://issues.apache.org/jira/browse/SPARK-36835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gengliang Wang resolved SPARK-36835. ------------------------------------ Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34085 [https://github.com/apache/spark/pull/34085] > Spark 3.2.0 POMs are no longer "dependency reduced" > --------------------------------------------------- > > Key: SPARK-36835 > URL: https://issues.apache.org/jira/browse/SPARK-36835 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 3.2.0 > Reporter: Josh Rosen > Assignee: Chao Sun > Priority: Blocker > Fix For: 3.2.0 > > > It looks like Spark 3.2.0's POMs are no longer "dependency reduced". As a > result, applications may pull in additional unnecessary dependencies when > depending on Spark. > Spark uses the Maven Shade plugin to create effective POMs and to bundle > shaded versions of certain libraries with Spark (namely, Jetty, Guava, and > JPPML). [By > default|https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#createDependencyReducedPom], > the Maven Shade plugin generates simplified POMs which remove dependencies > on artifacts that have been shaded. > SPARK-33212 / > [b6f46ca29742029efea2790af7fdefbc2fcf52de|https://github.com/apache/spark/commit/b6f46ca29742029efea2790af7fdefbc2fcf52de] > changed the configuration of the Maven Shade plugin, setting > {{createDependencyReducedPom}} to {{false}}. > As a result, the generated POMs now include compile-scope dependencies on the > shaded libraries. For example, compare the {{org.eclipse.jetty}} dependencies > in: > * Spark 3.1.2: > [https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/3.1.2/spark-core_2.12-3.1.2.pom] > * Spark 3.2.0 RC2: > [https://repository.apache.org/content/repositories/orgapachespark-1390/org/apache/spark/spark-core_2.12/3.2.0/spark-core_2.12-3.2.0.pom] > I think we should revert back to generating "dependency reduced" POMs to > ensure that Spark declares a proper set of dependencies and to avoid "unknown > unknown" consequences of changing our generated POM format. > /cc [~csun] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org