dongjoon-hyun commented on code in PR #60049:
URL: https://github.com/apache/spark/pull/60049#discussion_r4451488871
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala:
##########
@@ -261,9 +262,13 @@ object PROCESS_TABLES extends QueryTest {
val isPythonVersionAvailable = TestUtils.isPythonVersionAvailable
val releaseMirror = sys.env.getOrElse("SPARK_RELEASE_MIRROR",
"https://dist.apache.org/repos/dist/release";)
- // Tests the latest version of every release line if Java version is at most
17.
+ // Spark 3.3, 4.0, and 4.2 are the first release lines that support Java 17,
21, and 25,
+ // respectively. No release line supports the other Java versions, e.g.,
Java 26.
+ val minimumTestingVersion: Option[String] =
+ Map(17 -> "3.3.0", 21 -> "4.0.0", 25 ->
"4.2.0").get(Runtime.version().feature())
Review Comment:
Thank you. I added the note. Let me keep `assume` out of this PR because it
also changes how the existing network-failure path (SPARK-34508) is reported.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala:
##########
@@ -198,10 +198,11 @@ class HiveExternalCatalogVersionsSuite extends
SparkSubmitTestUtils {
if (PROCESS_TABLES.testingVersions.isEmpty) {
if (PROCESS_TABLES.isPythonVersionAvailable) {
- if (Utils.isJavaVersionAtMost17) {
+ if (PROCESS_TABLES.minimumTestingVersion.isDefined) {
logError("Fail to get the latest Spark versions to test.")
Review Comment:
Thank you. The new note asks to add an entry only after a Spark release
supports the Java version, so the fetched listing has at least that release.
Since this only affects the log message, I'll keep it.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala:
##########
@@ -273,6 +278,7 @@ object PROCESS_TABLES extends QueryTest {
.filter(sparkVersionPattern.unanchored.matches(_))
.map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
.filter(_ < org.apache.spark.SPARK_VERSION)
+ .filter(_ >= minimumTestingVersion.get)
Review Comment:
Thank you. I added `PROCESS_TABLES.javaVersion` and replaced `.get` with
`exists`, which keeps the existing `if` without re-indenting the block. For the
string comparison, I kept it consistent with `_ <
org.apache.spark.SPARK_VERSION` because `sparkVersionPattern` captures
single-digit components only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
