[
https://issues.apache.org/jira/browse/SPARK-54793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wy updated SPARK-54793:
-----------------------
Description:
When building Spark with Maven Shade plugin, some dependencies, like Jetty, are
relocated to some private packages like `org.sparkproject.jetty` in the final
JAR. However, Scala signatures remain unmodified while JVM bytecode is
relocated, causing binary compatibility breaks for downstream Scala projects.
This manifests as runtime linkage errors when Scala clients interact with
shaded APIs.
After building Spark, it can be confirmed by the following commands:
{code:java}
javap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar -verbose
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|descriptor' | grep 'public
synchronized void'
scalap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|ServletContextHandler' |
grep 'def attachHandler(handler'{code}
The outputs shows that the Scala signature is inconsistent with the binary:
{code:java}
2065: public synchronized void
attachHandler(org.sparkproject.jetty.servlet.ServletContextHandler);
20: def attachHandler(handler:
org.eclipse.jetty.servlet.ServletContextHandler): scala.Unit = { /* compiled
code */ }
{code}
This will break the downstream projects who use Spark as a dependency.
was:
When building Spark with Maven Shade plugin, some dependencies, like Jetty, are
relocated to some private packages like `org.sparkproject.jetty` in the final
JAR. However, Scala signatures remain unmodified while JVM bytecode is
relocated, causing binary compatibility breaks for downstream Scala projects.
This manifests as runtime linkage errors when Scala clients interact with
shaded APIs.
After building Spark, it can be confirmed by the following commands:
{code:java}
javap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar -verbose
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|descriptor' | grep 'public
synchronized void'
scalap -classpath
./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar
org.apache.spark.ui.WebUI | grep -n 'attachHandler\|ServletContextHandler' |
grep 'def attachHandler(handler'{code}
The outputs shows that the Scala signature is inconsistent with the binary:
{code:java}
2065: public synchronized void
attachHandler(org.sparkproject.jetty.servlet.ServletContextHandler);
20: def attachHandler(handler:
org.eclipse.jetty.servlet.ServletContextHandler): scala.Unit = { /* compiled
code */ }
{code}
This will break the downstream projects who use Spark as a dependency. Here's a
small repro:
[kuwii/spark-shade-repro|https://github.com/kuwii/spark-shade-repro].
> Shade relocation breaks Scala binary compatibility due to unmodified Scala
> signatures
> -------------------------------------------------------------------------------------
>
> Key: SPARK-54793
> URL: https://issues.apache.org/jira/browse/SPARK-54793
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.1.0, 4.2.0
> Reporter: wy
> Priority: Minor
>
> When building Spark with Maven Shade plugin, some dependencies, like Jetty,
> are relocated to some private packages like `org.sparkproject.jetty` in the
> final JAR. However, Scala signatures remain unmodified while JVM bytecode is
> relocated, causing binary compatibility breaks for downstream Scala projects.
> This manifests as runtime linkage errors when Scala clients interact with
> shaded APIs.
> After building Spark, it can be confirmed by the following commands:
>
> {code:java}
> javap -classpath
> ./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar -verbose
> org.apache.spark.ui.WebUI | grep -n 'attachHandler\|descriptor' | grep
> 'public synchronized void'
> scalap -classpath
> ./assembly/target/scala-2.13/jars/spark-core_2.13-4.2.0-SNAPSHOT.jar
> org.apache.spark.ui.WebUI | grep -n 'attachHandler\|ServletContextHandler' |
> grep 'def attachHandler(handler'{code}
> The outputs shows that the Scala signature is inconsistent with the binary:
> {code:java}
> 2065: public synchronized void
> attachHandler(org.sparkproject.jetty.servlet.ServletContextHandler);
> 20: def attachHandler(handler:
> org.eclipse.jetty.servlet.ServletContextHandler): scala.Unit = { /* compiled
> code */ }
> {code}
>
> This will break the downstream projects who use Spark as a dependency.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
