This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c12eb2c93d7 [SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to
4.7.1
c12eb2c93d7 is described below
commit c12eb2c93d7678795726a7f2ccbb30a66a93fb22
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Wed Sep 13 08:43:52 2023 -0700
[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1
### What changes were proposed in this pull request?
This pr downgrade `scala-maven-plugin` to version 4.7.1 to avoid it
automatically adding the `-release` option as a Scala compilation argument.
### Why are the changes needed?
The `scala-maven-plugin` versions 4.7.2 and later will try to automatically
append the `-release` option as a Scala compilation argument when it is not
specified by the user:
1. 4.7.2 and 4.8.0: try to add the `-release` option for Scala versions
2.13.9 and higher.
2. 4.8.1: try to append the `-release` option for Scala versions
2.12.x/2.13.x/3.1.1, and append `-java-output-version` for Scala 3.1.2.
The addition of the `-release` option has caused issues mentioned in
SPARK-44376 | https://github.com/apache/spark/pull/41943 and
https://github.com/apache/spark/pull/40442#issuecomment-1474161688. This is
because the `-release` option has stronger compilation restrictions than
`-target`, ensuring not only bytecode format, but also that the API used in the
code is compatible with the specified version of Java. However, many APIs in
the `sun.*` package are not `exports` in Java 11, 17, [...]
For discussions within the Scala community, see
https://github.com/scala/bug/issues/12643,
https://github.com/scala/bug/issues/12824,
https://github.com/scala/bug/issues/12866, but this is not a bug.
I have also submitted an issue to the `scala-maven-plugin` community to
discuss the possibility of adding additional settings to control the addition
of the `-release` option:
https://github.com/davidB/scala-maven-plugin/issues/722.
For Apache Spark 4.0, in the short term, I suggest downgrading
`scala-maven-plugin` to version 4.7.1 to avoid it automatic adding the
`-release` option as a Scala compilation argument. In the long term, we should
reduce use of APIs that are not `exports` for compatibility with the `-release`
compilation option due to `-target` already deprecated after Scala 2.13.9.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual check
run `git revert 656bf36363c466b60d00452399994ccaaa654ed8` to revert to
using Scala 2.13.11 and run `dev/change-scala-version.sh 2.13` to change Scala
to 2.13
1. Run `build/mvn clean install -DskipTests -Pscala-2.13 -X` to check the
Scala compilation arguments.
Before
```
[[DEBUG] [zinc] Running cached compiler 1992eaf4 for Scala compiler version
2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
-unchecked
-deprecation
-feature
-explaintypes
-target:jvm-1.8
-Wconf:cat=deprecation:wv,any:e
-Wunused:imports
-Wconf:cat=scaladoc:wv
-Wconf:cat=lint-multiarg-infix:wv
-Wconf:cat=other-nullary-override:wv
-Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local
OutputCommitCoordinatorSuite>.futureAction:wv
-Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since
2.13).+$:s
-Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it
loses precision).+$:s
-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
-Wconf:msg=method with a single empty parameter list overrides method
without any parameter list:s
-Wconf:msg=method without a parameter list overrides a method with a
single empty one:s
-Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
-Wconf:cat=unchecked&msg=outer reference:s
-Wconf:cat=unchecked&msg=eliminated by erasure:s
-Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
-Wconf:msg=Implicit definition should have explicit type:s
-release
8
-bootclasspath
...
```
After
```
[DEBUG] [zinc] Running cached compiler 72dd4888 for Scala compiler version
2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
-unchecked
-deprecation
-feature
-explaintypes
-target:jvm-1.8
-Wconf:cat=deprecation:wv,any:e
-Wunused:imports
-Wconf:cat=scaladoc:wv
-Wconf:cat=lint-multiarg-infix:wv
-Wconf:cat=other-nullary-override:wv
-Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
-Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local
OutputCommitCoordinatorSuite>.futureAction:wv
-Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since
2.13).+$:s
-Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it
loses precision).+$:s
-Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
-Wconf:msg=method with a single empty parameter list overrides method
without any parameter list:s
-Wconf:msg=method without a parameter list overrides a method with a
single empty one:s
-Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
-Wconf:cat=unchecked&msg=outer reference:s
-Wconf:cat=unchecked&msg=eliminated by erasure:s
-Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
-Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
-Wconf:msg=Implicit definition should have explicit type:s
-target:8
-bootclasspath
...
```
After downgrading the version, the `-release` option should no longer
appear in the compilation arguments.
2. Maven can build the project with Java 17 without the issue described in
https://github.com/apache/spark/pull/41943. And after this pr, we can
re-upgrade Scala 2.13 to Scala 2.13.11.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #42899 from LuciferYang/SPARK-45144.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pom.xml b/pom.xml
index 02920c0ae74..0ebd31ebe33 100644
--- a/pom.xml
+++ b/pom.xml
@@ -175,8 +175,8 @@
<scala.version>2.12.18</scala.version>
<scala.binary.version>2.12</scala.binary.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
- <!-- don't upgrade scala-maven-plugin to version 4.8.1 SPARK-42809 and
SPARK-43595 -->
- <scala-maven-plugin.version>4.8.0</scala-maven-plugin.version>
+ <!-- don't upgrade scala-maven-plugin to version 4.7.2 or higher, see
SPARK-45144 for details -->
+ <scala-maven-plugin.version>4.7.1</scala-maven-plugin.version>
<maven.scaladoc.skip>false</maven.scaladoc.skip>
<versions-maven-plugin.version>2.16.0</versions-maven-plugin.version>
<!-- for now, not running scalafmt as part of default verify pipeline -->
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
