This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 1cfa2624c93 [SPARK-44690][SPARK-44376][BUILD] Downgrade Scala to 2.13.8
1cfa2624c93 is described below
commit 1cfa2624c933088deef4281b207631c806230440
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Mon Aug 7 10:13:59 2023 +0800
[SPARK-44690][SPARK-44376][BUILD] Downgrade Scala to 2.13.8
### What changes were proposed in this pull request?
The aim of this PR is to downgrade the Scala 2.13 dependency to 2.13.8 to
ensure that Spark can be build with `-target:jvm-1.8`, and tested with Java
11/17.
### Why are the changes needed?
As reported in SPARK-44376, there are issues when maven build and test
using Java 11/17 with `-target:jvm-1.8`:
- run `build/mvn clean install -Pscala-2.13` with Java 17
```
[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first)
spark-core_2.13 ---
[INFO] Compiler bridge file:
/Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to
/Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation | origin= | version=] -target is
deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71:
not found: value sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210:
not found: type Unsafe
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212:
not found: type Unsafe
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452:
not found: value sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99:
not found: type SignalHandler
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108:
not found: type SignalHandler
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
Unused import
[WARNING] one warning found
[ERROR] 23 errors found
```
- run `build/mvn clean install -Pscala-2.13 -Djava.version=17` with Java 17
```
[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first)
spark-tags_2.13 ---
[INFO] Compiler bridge file:
/Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 2 Scala sources and 8 Java sources to
/Users/yangjie01/SourceCode/git/spark-mine-13/common/tags/target/scala-2.13/classes
...
[WARNING] [Warn] : [deprecation | origin= | version=] -target is
deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] : target platform version 8 is older than the release
version 17
[WARNING] one warning found
[ERROR] one error found
```
- run `build/mvn clean package -Pscala-2.13 -DskipTests` or `build/mvn
clean install -Pscala-2.13 -DskipTests` with Java 8 first, then run `build/mvn
test -Pscala-2.13` with Java 17
```
[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first)
spark-core_2.13 ---
[INFO] Compiler bridge file:
/Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to
/Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation | origin= | version=] -target is
deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71:
not found: value sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210:
not found: type Unsafe
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212:
not found: type Unsafe
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236:
not found: type DirectBuffer
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452:
not found: value sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
not found: object sun
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99:
not found: type SignalHandler
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108:
not found: type SignalHandler
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114:
not found: type Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128:
not found: value Signal
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
Unused import
[ERROR] [Error]
/Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26:
Unused import
[WARNING] one warning found
[ERROR] 23 errors found
```
This is inconsistent with the behavior of the released `Apache Spark`
version, so we need to use the previous Scala2.13 version to support this
behavior.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass GitHub Actions
- Manual checked, the above command can run normally after this pr
Closes #42364 from LuciferYang/SPARK-44690.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: yangjie01 <yangji...@baidu.com>
(cherry picked from commit 656bf36363c466b60d00452399994ccaaa654ed8)
Signed-off-by: yangjie01 <yangji...@baidu.com>
---
pom.xml | 6 +-----
project/SparkBuild.scala | 4 +---
2 files changed, 2 insertions(+), 8 deletions(-)
diff --git a/pom.xml b/pom.xml
index d60722e7312..9438a112c24 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3600,7 +3600,7 @@
<profile>
<id>scala-2.13</id>
<properties>
- <scala.version>2.13.11</scala.version>
+ <scala.version>2.13.8</scala.version>
<scala.binary.version>2.13</scala.binary.version>
</properties>
<build>
@@ -3659,10 +3659,6 @@
-->
<arg>-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s</arg>
<arg>-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s</arg>
- <!--
- SPARK-40497 Upgrade Scala to 2.13.11 and suppress
`Implicit definition should have explicit type`
- -->
- <arg>-Wconf:msg=Implicit definition should have explicit
type:s</arg>
</args>
<compilerPlugins combine.self="override">
</compilerPlugins>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index e585d5dd2b2..79007626026 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -286,9 +286,7 @@ object SparkBuild extends PomBuild {
// TODO(SPARK-43850): Remove the following suppression rules and
remove `import scala.language.higherKinds`
// from the corresponding files when Scala 2.12 is no longer
supported.
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBase.scala:s",
-
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBaseOps.scala:s",
- // SPARK-40497 Upgrade Scala to 2.13.11 and suppress `Implicit
definition should have explicit type`
- "-Wconf:msg=Implicit definition should have explicit type:s"
+
"-Wconf:cat=unused-imports&src=org\\/apache\\/spark\\/graphx\\/impl\\/VertexPartitionBaseOps.scala:s"
)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
