andygrove opened a new issue, #6283:
URL: https://github.com/apache/datafusion-comet/issues/6283
### Describe the bug
The Spark 3.4 and 3.5 jars that Comet has published since 0.11.0 do not load
on Java 11, although the user guide for each of those releases lists Java 11 as
supported for Spark 3.4 and 3.5. The 1.0 guide says "Spark 3.5.9 is supported
with Java 11/17" and deprecates JDK 11 for removal in 1.1.0, so 1.0.x users on
Java 11 have no published jar that works.
In every Spark 3.x jar of 1.0.0, the 631 classes compiled by javac
(`NativeBase`, `CometVector`, `CometShuffleBlockIterator`, ...) have class file
version 61 (Java 17). The classes compiled by scalac are version 61 in the
Scala 2.13 jars and 52 in the Scala 2.12 jars. Every 1.0.0 manifest says
`Build-Jdk-Spec: 17`. The 0.10.0 `comet-spark-spark3.5_2.12` jar was built on
JDK 11 and is version 55, while 0.11.0 and 0.17.1 are version 61.
### Steps to reproduce
With Java 11 and the jars from Maven Central:
```
$ java -cp comet-spark-spark3.5_2.13-1.0.0.jar org.apache.spark.CometPlugin
Error: LinkageError occurred while loading main class
org.apache.spark.CometPlugin
java.lang.UnsupportedClassVersionError: org/apache/spark/CometPlugin
has been compiled by a more recent version of the Java Runtime (class file
version 61.0), this version of the Java Runtime only recognizes class file
versions up to 55.0
$ java -cp comet-spark-spark3.5_2.12-1.0.0.jar org.apache.comet.NativeBase
Error: LinkageError occurred while loading main class
org.apache.comet.NativeBase
java.lang.UnsupportedClassVersionError: org/apache/comet/NativeBase has
been compiled by a more recent version of the Java Runtime (class file version
61.0), this version of the Java Runtime only recognizes class file versions up
to 55.0
```
In a Spark application it fails as soon as the first such class loads. That
is how I found it: the Iceberg 1.8 job on `branch-1.0` (Spark 3.4.3, Java 11)
resolved the published 1.0.0 jar from Maven Central instead of the one it
built, and every test class failed with `UnsupportedClassVersionError` on
`CometShuffleManager` (#6277).
### Expected behavior
The Spark 3.4 and 3.5 jars target Java 11 (class file version 55), as the
`java.version=11` in the `spark-3.4` and `spark-3.5` profiles intends, and load
on every Java version the docs list.
### Additional context
#2514 made `dev/release/build-release-comet.sh` require Java 17, because the
Spark 4.0 build needs it, and the script builds every profile on that one JDK.
In the root pom, the `jdk17` profile is activated by the running JDK and is
declared after `spark-3.4` and `spark-3.5`, so its `java.version=17` wins over
theirs. With `branch-1.0`'s pom:
```
$ ./mvnw -N -q help:evaluate -Pspark-3.5 -Dexpression=maven.compiler.target
-DforceStdout # on JDK 11
11
$ ./mvnw -N -q help:evaluate -Pspark-3.5 -Dexpression=maven.compiler.target
-DforceStdout # on JDK 17
17
```
`main` drops Java 11 in 1.1.0, so this only matters for the 1.0.x line,
where a 1.0.1 built by the same script would ship the same jars (see #6201).
The published releases can't be changed. For 1.0.1, the Spark 3.x jars could be
built on JDK 11, or the Spark profile's `java.version` could win over the JDK
profile's, preferably with `--release 11` so that javac also checks the API. A
class file version check on the Spark 3.x jars in
`dev/release/verify-release-candidate.sh` would keep this from coming back, and
the 1.0 docs could say that the published 0.11.0 to 1.0.0 jars need Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
