wy created SPARK-54793:
--------------------------
Summary: Shade relocation breaks Scala binary compatibility due to
unmodified Scala signatures
Key: SPARK-54793
URL: https://issues.apache.org/jira/browse/SPARK-54793
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.1.0, 4.2.0
Reporter: wy
When building Spark with Maven Shade plugin, some dependencies, like Jetty, are
relocated to some private packages like `org.sparkproject.jetty` in the final
JAR. However, Scala signatures remain unmodified while JVM bytecode is
relocated, causing binary compatibility breaks for downstream Scala projects.
This manifests as runtime linkage errors when Scala clients interact with
shaded APIs.
After building Spark, it can be confirmed by the following commands:
{code:java}
javap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar -verbose
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|descriptor' | grep 'public
synchronized void'
scalap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|ServletContextHandler' |
grep 'def attachHandler(handler'{code}
The outputs shows that the Scala signature is inconsistent with the binary:
{code:java}
2065: public synchronized void
attachHandler(org.sparkproject.jetty.servlet.ServletContextHandler);
20: def attachHandler(handler:
org.eclipse.jetty.servlet.ServletContextHandler): scala.Unit = { /* compiled
code */ }
{code}
This will break the downstream projects who use Spark as a dependency. Here's a
small repro:
[kuwii/spark-shade-repro|https://github.com/kuwii/spark-shade-repro].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
