This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8332f0b6d33 [SPARK-44687][BUILD] Fix mima check for Scala 2.13 after
SPARK-44198 merged
8332f0b6d33 is described below
commit 8332f0b6d33ade037e65459aced47e18fb41f76c
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Sun Aug 6 01:13:43 2023 +0800
[SPARK-44687][BUILD] Fix mima check for Scala 2.13 after SPARK-44198 merged
### What changes were proposed in this pull request?
This pr aims add a new `ProblemFilters` to `MimaExcludes.scala` to fix mima
check for Scala 2.13 after SPARK-44198 merged.
### Why are the changes needed?
Scala 2.13's daily tests have been failing the mima check for several days:
- https://github.com/apache/spark/actions/runs/5765663964
<img width="1194" alt="image"
src="https://github.com/apache/spark/assets/1475305/7b73aa0d-3e19-4119-bdbe-627f1c715d2b";>
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual verification：
1. The mima check was passing before SPARK-44198.
```
// [SPARK-44425][CONNECT] Validate that user provided sessionId is an UUID
git reset --hard a3bd477a6d8c317ee1e9a6aae6ebd2ef4fc67cce
dev/change-scala-version.sh 2.13
dev/mima -Pscala-2.13
```
```
[success] Total time: 129 s (02:09), completed 2023-8-5 14:21:06
```
2. The mima check failed after SPARK-44198 was merged
```
// [SPARK-44198][CORE] Support propagation of the log level to the executors
git reset --hard 5fc90fbd4e3235fbcf038f4725037321b8234d94
dev/change-scala-version.sh 2.13
dev/mima -Pscala-2.13
```
```
[error] spark-core: Failed binary compatibility check against
org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered
4013)
[error] * the type hierarchy of object
org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages#SparkAppConfig
is different in current version. Missing types {scala.runtime.AbstractFunction4}
[error] filter with:
ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$SparkAppConfig$")
[error] java.lang.RuntimeException: Failed binary compatibility check
against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems
(filtered 4013)
[error] at scala.sys.package$.error(package.scala:30)
[error] at
com.typesafe.tools.mima.plugin.SbtMima$.reportModuleErrors(SbtMima.scala:89)
[error] at
com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2(MimaPlugin.scala:36)
[error] at
com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2$adapted(MimaPlugin.scala:26)
[error] at scala.collection.Iterator.foreach(Iterator.scala:943)
[error] at scala.collection.Iterator.foreach$(Iterator.scala:943)
[error] at
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
[error] at
com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1(MimaPlugin.scala:26)
[error] at
com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1$adapted(MimaPlugin.scala:25)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at
sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] at
sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] at sbt.Execute.work(Execute.scala:292)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] at
sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] at
sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:750)
[error] (core / mimaReportBinaryIssues) Failed binary compatibility check
against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems
(filtered 4013)
[error] Total time: 82 s (01:22), completed 2023-8-5 14:23:49
```
3. with this pr, mima check pass
```
gh pr checkout 42358
dev/change-scala-version.sh 2.13
dev/mima -Pscala-2.13
```
```
[success] Total time: 157 s (02:37), completed 2023-8-5 14:31:05
```
Closes #42358 from LuciferYang/SPARK-44687.
Authored-by: yangjie01 <yangji...@baidu.com>
Signed-off-by: yangjie01 <yangji...@baidu.com>
---
project/MimaExcludes.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index 14fa43b5672..d0fc8f2b116 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -73,7 +73,9 @@ object MimaExcludes {
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.AnalysisException$"),
// [SPARK-44535][CONNECT][SQL] Move required Streaming API to sql/api
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.streaming.GroupStateTimeout"),
-
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.streaming.OutputMode")
+
ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.streaming.OutputMode"),
+ // [SPARK-44198][CORE] Support propagation of the log level to the
executors
+
ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$SparkAppConfig$")
)
// Default exclude rules
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
