Yosef Fertel created SPARK-54049:
------------------------------------
Summary: spark-network-common no longer shades all of Guava
Key: SPARK-54049
URL: https://issues.apache.org/jira/browse/SPARK-54049
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 4.0.0
Reporter: Yosef Fertel
Shading of Guava in `spark-network-common` has changed from 3.x to 4.x such
that conflicting classes prevent downstream code from using Guava.
In spark jars prior to 4.x,
`com/google/thirdparty/publicsuffix/PublicSuffixPatterns.class` was not a class
included in the jar. Post 4.x, Guava is still shaded to `sparkproject` but
`com/google/thirdparty/publicsuffix/PublicSuffixPatterns.class`, used in
Guava's `InternetDomainName` and elsewhere, is now present in the jar.
{code:java}
$ jar tf spark-network-common_2.13-4.0.0.jar | rg PublicSuffixPatterns
com/google/thirdparty/publicsuffix/PublicSuffixPatterns.class {code}
If a library depends on spark but also depends on Guava, calls to
`InternetDomainName` that call `PublicSuffixPatterns` will fail with `Exception
in thread "main" java.lang.NoSuchFieldError: EXACT`.
Inspecting the code locations in such a library via
`classOf[InternetDomainName].getProtectionDomain.getCodeSource.getLocation` and
`classOf[PublicSuffixPatterns].getProtectionDomain.getCodeSource.getLocation`
reveals that `InternetDomainName` is sourced from Guava,
`target/bg-jobs/sbt_2062a9c3/target/5fcb43b5/1685140132000/guava-32.0.0-jre.jar`,
while `PublicSuffixPatterns` is sourced instead from spark jar,
`target/bg-jobs/sbt_2062a9c3/target/db746978/1747651686000/spark-network-common_2.13-4.0.0.jar`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
