dongjoon-hyun opened a new pull request, #60049: URL: https://github.com/apache/spark/pull/60049
### What changes were proposed in this pull request? This PR aims to make `HiveExternalCatalogVersionsSuite` test the old Spark versions that support the running Java version, instead of skipping all of them on Java 21 and later. | Java | Release lines to test | Old Spark versions to test today | | :-: | :-: | :-: | | 17 | 3.3 and later (no change) | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | | 21 | 4.0 and later | 4.0.5, 4.1.4, 4.2.1, 4.3.0 | | 25 | 4.2 and later | 4.2.1, 4.3.0 | | Others, e.g., 26 | None (no change) | None | Each minimum release line is the first release line whose documentation lists the Java version. - Apache Spark 3.3 to 3.5: `Spark runs on Java 8/11/17` - Apache Spark 4.0 and 4.1: `Spark runs on Java 17/21` - Apache Spark 4.2 and 4.3: `Spark runs on Java 17/21/25` No release line supports the other Java versions, e.g., Java 26. So, this suite keeps skipping the old Spark versions on them, and the log message shows the running Java version instead of `Java 21`. In addition, this PR uses `Runtime.version().feature()` instead of `Utils.isJavaVersionAtMost17`. ### Why are the changes needed? SPARK-44095 (Apache Spark 3.5.0) made this suite skip the old Spark versions on Java 21 because no Spark release supported Java 21 at that time. Now, Apache Spark 4.0.0 and later support Java 21, and Apache Spark 4.2.0 and later support Java 25. However, the Java 21 and Java 25 CIs still skip all old Spark versions. Moreover, SPARK-60669 (#59897) is going to drop Java 17 support at Apache Spark 5.0.0. After that, no `master` CI runs this suite on Java 17, and `master` would lose this backward compatibility test completely. ### Does this PR introduce _any_ user-facing change? No. This is a test-only change. ### How was this patch tested? Pass the CIs. In addition, I ran this suite manually with Java 21, 25, and 17. ``` $ build/sbt -Phive 'hive/testOnly *HiveExternalCatalogVersionsSuite' ... [info] HiveExternalCatalogVersionsSuite: [info] - backward compatibility (16 seconds, 636 milliseconds) [info] Run completed in 52 seconds, 578 milliseconds. [info] Total number of tests run: 1 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. $ grep -E 'SparkContext: (Running Spark version|Java version)' sql/hive/target/unit-tests.log | sed 's/.*SparkContext: //' Running Spark version 4.0.5 Java version 21.0.13 Running Spark version 4.1.4 Java version 21.0.13+9-LTS Running Spark version 4.2.1 Java version 21.0.13+9-LTS Running Spark version 4.3.0 Java version 21.0.13+9-LTS ``` | Java | Old Spark versions that ran `spark-submit` | `backward compatibility` | | :-: | :-: | :-: | | 17.0.21 | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (17 seconds) | | 21.0.13 | 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) | | 25.0.5 | 4.2.1, 4.3.0 | Passed (13 seconds) | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
