This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9785353684b [SPARK-43537][INFA][BUILD] Upgrading the ASM dependencies
used in the `tools` module to 9.4
9785353684b is described below
commit 9785353684bdc2a2c7445b7e6b9ab85154f6933f
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Wed May 17 11:18:14 2023 -0500
[SPARK-43537][INFA][BUILD] Upgrading the ASM dependencies used in the
`tools` module to 9.4
### What changes were proposed in this pull request?
This pr aims upgrade ASM related dependencies in the `tools` module from
version 7.1 to version 9.4 to make `GenerateMIMAIgnore` can process Java 17+
compiled code.
Additionally, this pr defines `asm.version` to manage versions of ASM.
### Why are the changes needed?
The classpath processed by `GenerateMIMAIgnore` cannot contain Java 17+
compiled code now due to the ASM version use by `tools` module is too low, but
https://github.com/bmc/classutil has not been updated for a long time, we can't
solve the problem by upgrading `classutil`, so this pr make the `tools` module
explicitly rely on ASM 9.4 for workaround.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Action
- Manual checked `dev/mima` due to this pr upgrade the dependency of tools
module
```
dev/mima
```
and
```
dev/change-scala-version.sh 2.13
dev/mima -Pscala-2.13
```
- A case that can reproduce the problem: run following script with master
branch:
```
set -o pipefail
set -e
FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
cd "$FWDIR"
export SPARK_HOME=$FWDIR
echo $SPARK_HOME
if [[ -x "$JAVA_HOME/bin/java" ]]; then
JAVA_CMD="$JAVA_HOME/bin/java"
else
JAVA_CMD=java
fi
TOOLS_CLASSPATH="$(build/sbt -DcopyDependencies=false "export
tools/fullClasspath" | grep jar | tail -n1)"
ASSEMBLY_CLASSPATH="$(build/sbt -DcopyDependencies=false "export
assembly/fullClasspath" | grep jar | tail -n1)"
rm -f .generated-mima*
$JAVA_CMD \
-Xmx2g \
-XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.util.jar=ALL-UNNAMED \
-cp "$TOOLS_CLASSPATH:$ASSEMBLY_CLASSPATH" \
org.apache.spark.tools.GenerateMIMAIgnore
rm -f .generated-mima*
```
**Before**
```
Exception in thread "main" java.lang.IllegalArgumentException: Unsupported
class file major version 61
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:195)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:176)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:162)
at org.objectweb.asm.ClassReader.<init>(ClassReader.java:283)
at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:222)
at org.clapper.classutil.ClassFinder.classData(ClassFinder.scala:404)
at
org.clapper.classutil.ClassFinder.$anonfun$processOpenZip$2(ClassFinder.scala:359)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator.toStream(Iterator.scala:1417)
at scala.collection.Iterator.toStream$(Iterator.scala:1416)
at scala.collection.AbstractIterator.toStream(Iterator.scala:1431)
at scala.collection.Iterator.$anonfun$toStream$1(Iterator.scala:1417)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at
scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at
scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at scala.collection.immutable.Stream.$anonfun$map$1(Stream.scala:418)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at scala.collection.immutable.Stream.filterImpl(Stream.scala:506)
at
scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at
scala.collection.immutable.Stream$.$anonfun$filteredTail$1(Stream.scala:1260)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
at scala.collection.generic.Growable.loop$1(Growable.scala:57)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:61)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
at
scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:381)
at
scala.collection.immutable.Set$SetBuilderImpl.$plus$plus$eq(Set.scala:329)
at scala.collection.TraversableLike.to(TraversableLike.scala:786)
at scala.collection.TraversableLike.to$(TraversableLike.scala:783)
at scala.collection.AbstractTraversable.to(Traversable.scala:108)
at scala.collection.TraversableOnce.toSet(TraversableOnce.scala:360)
at scala.collection.TraversableOnce.toSet$(TraversableOnce.scala:360)
at scala.collection.AbstractTraversable.toSet(Traversable.scala:108)
at
org.apache.spark.tools.GenerateMIMAIgnore$.getClasses(GenerateMIMAIgnore.scala:156)
at
org.apache.spark.tools.GenerateMIMAIgnore$.privateWithin(GenerateMIMAIgnore.scala:57)
at
org.apache.spark.tools.GenerateMIMAIgnore$.main(GenerateMIMAIgnore.scala:122)
at
org.apache.spark.tools.GenerateMIMAIgnore.main(GenerateMIMAIgnore.scala)
```
The script run failed without this pr due to `ASSEMBLY_CLASSPATH` contains
jackson-core 2.15.0 and it contains code compiled from Java 17+
<img width="463" alt="image"
src="https://github.com/apache/spark/assets/1475305/85b6b269-a182-460a-9995-7b1a517c2dfe";>
**After**
No more errors like `Exception in thread "main"
java.lang.IllegalArgumentException: Unsupported class file major version 61`
with this pr.
Closes #41198 from LuciferYang/tools-asm.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Sean Owen <sro...@gmail.com>
---
pom.xml | 5 +++--
tools/pom.xml | 21 +++++++++++++++++++++
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index c0506d7e31d..a81e6df057d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -117,6 +117,7 @@
<maven.version>3.8.8</maven.version>
<exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
<sbt.project.name>spark</sbt.project.name>
+ <asm.version>9.4</asm.version>
<slf4j.version>2.0.7</slf4j.version>
<log4j.version>2.20.0</log4j.version>
<!-- make sure to update IsolatedClientLoader whenever this version is
changed -->
@@ -3130,12 +3131,12 @@
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
- <version>9.4</version>
+ <version>${asm.version}</version>
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
- <version>9.4</version>
+ <version>${asm.version}</version>
</dependency>
</dependencies>
</plugin>
diff --git a/tools/pom.xml b/tools/pom.xml
index ce2aebe9e59..9dc136d4268 100644
--- a/tools/pom.xml
+++ b/tools/pom.xml
@@ -44,8 +44,29 @@
<dependency>
<groupId>org.clapper</groupId>
<artifactId>classutil_${scala.binary.version}</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
<version>1.5.1</version>
</dependency>
+ <dependency>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>asm</artifactId>
+ <version>${asm.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>asm-commons</artifactId>
+ <version>${asm.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>asm-util</artifactId>
+ <version>${asm.version}</version>
+ </dependency>
</dependencies>
<build>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
