This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new e5452c6ff38d feat(spark): upgrade Spark 4.2 dependency to GA 4.2.0
(#19365)
e5452c6ff38d is described below
commit e5452c6ff38d71da6cf43a9f664e61e6e4aadc9b
Author: Y Ethan Guo <[email protected]>
AuthorDate: Fri Jul 24 01:57:35 2026 -0700
feat(spark): upgrade Spark 4.2 dependency to GA 4.2.0 (#19365)
* chore(spark): upgrade Spark 4.2 dependency to GA 4.2.0
Bump the spark4.2 profile from 4.2.0-preview4 to the official 4.2.0 GA
release. Align the transitive versions that changed in GA: hadoop
3.4.3 -> 3.5.0 and lz4-java 1.10.4 -> 1.11.0. Rename the CI bundle
validation base image tag flink1200hive313spark420previewscala213 ->
flink1200hive313spark420scala213, rebuild it with SPARK_VERSION=4.2.0
and HADOOP_VERSION=3.5.0, and update the Spark 4.2 runtime string to
spark4.2.0 across the CI scripts, bot.yml, and .asf.yaml required checks.
closes #19364
* Adapt hudi-spark4 modules to Spark 4.2.0 GA Catalyst API changes
Spark 4.2.0 GA made three source-breaking Catalyst changes relative to
4.2.0-preview4:
- TableOutputResolver.resolveOutputColumns replaced the boolean
supportColDefaultValue flag with a defaultValueFillMode enum. Moved the
call into a per-version resolveOutputColumns seam on the shared
hudi-spark4-common base: 4.0/4.1 pass supportColDefaultValue = true, 4.2
passes defaultValueFillMode = FILL (the exact equivalent of the old true).
- Removed the geospatial GeographyVal/GeometryVal types and
SpecializedGetters.getGeography/getGeometry. Dropped the corresponding
overrides from the Spark 4.2 InternalRow and partition-values wrappers.
- Added SpecializedGetters.getBinaryView(int): BinaryView. Implemented it on
the Spark 4.2 wrappers; the partition-mapping subclasses inherit it.
* Move .asf.yaml required-check updates to a separate PR
The .asf.yaml required_status_checks rename (spark4.2.0-preview4 ->
spark4.2.0) must land before this PR so the renamed Spark 4.2 jobs can
satisfy the required checks. It is handled in its own PR; revert the
.asf.yaml change here to keep this PR free of it.
* chore(spark): address review comments on Spark 4.2 GA upgrade
- validate.sh: collapse the three per-version Spark 4 checks to a
`[[ "$SPARK_VERSION" == 4.* ]]` glob so future Spark 4 bumps don't
touch this file.
- release_candidate_validation.yml: add the spark4.2 / spark4.2.0 entry
to validate-release-candidate-bundles-spark4 so the
hudi-spark4.2-bundle_2.13 gets bundle-validated at RC time.
---
.github/workflows/bot.yml | 4 ++--
.github/workflows/release_candidate_validation.yml | 4 ++++
.../sql/hudi/analysis/HoodieSpark4Analysis.scala | 22 ++++++++++++++++------
.../sql/hudi/analysis/HoodieSpark40Analysis.scala | 9 +++++++++
.../sql/hudi/analysis/HoodieSpark41Analysis.scala | 9 +++++++++
.../apache/hudi/Spark42HoodiePartitionValues.scala | 10 +++-------
.../client/model/Spark42HoodieInternalRow.scala | 13 ++++---------
.../sql/hudi/analysis/HoodieSpark42Analysis.scala | 10 ++++++++++
...h => build_flink1200hive313spark420scala213.sh} | 8 ++++----
packaging/bundle-validation/ci_run.sh | 8 ++++----
packaging/bundle-validation/run_docker_java17.sh | 8 ++++----
packaging/bundle-validation/validate.sh | 6 +++---
pom.xml | 6 +++---
13 files changed, 75 insertions(+), 42 deletions(-)
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index ba93eb03db30..3f9f7eac868c 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -1176,7 +1176,7 @@ jobs:
- scalaProfile: 'scala-2.13'
flinkProfile: 'flink1.20'
sparkProfile: 'spark4.2'
- sparkRuntime: 'spark4.2.0-preview4'
+ sparkRuntime: 'spark4.2.0'
steps:
- if: needs.changes.outputs.relevant == 'true'
@@ -1304,7 +1304,7 @@ jobs:
flinkAvroVersion: '1.11.4'
flinkParquetVersion: '1.13.1'
sparkProfile: 'spark4.2'
- sparkRuntime: 'spark4.2.0-preview4'
+ sparkRuntime: 'spark4.2.0'
steps:
- if: needs.changes.outputs.relevant == 'true'
uses: actions/checkout@v5
diff --git a/.github/workflows/release_candidate_validation.yml
b/.github/workflows/release_candidate_validation.yml
index 6d13467ce87f..cb8a976f6b47 100644
--- a/.github/workflows/release_candidate_validation.yml
+++ b/.github/workflows/release_candidate_validation.yml
@@ -82,6 +82,10 @@ jobs:
flinkProfile: 'flink1.20'
sparkProfile: 'spark4.1'
sparkRuntime: 'spark4.1.1'
+ - scalaProfile: 'scala-2.13'
+ flinkProfile: 'flink1.20'
+ sparkProfile: 'spark4.2'
+ sparkRuntime: 'spark4.2.0'
steps:
- uses: actions/checkout@v5
- name: Set up JDK 17
diff --git
a/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark4Analysis.scala
b/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark4Analysis.scala
index 8d4266011183..85f03752da95 100644
---
a/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark4Analysis.scala
+++
b/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark4Analysis.scala
@@ -18,9 +18,10 @@
package org.apache.spark.sql.hudi.analysis
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.analysis.{ResolveInsertionBase,
TableOutputResolver}
+import org.apache.spark.sql.catalyst.analysis.ResolveInsertionBase
import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement,
LogicalPlan}
import org.apache.spark.sql.errors.DataTypeErrors.toSQLId
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.datasources.PreprocessTableInsertion
@@ -55,6 +56,17 @@ import
org.apache.spark.sql.util.PartitioningUtils.normalizePartitionSpec
*/
abstract class HoodieSpark4ResolveColumnsForInsertInto extends
ResolveInsertionBase {
+ /**
+ * Resolves the query output against the table schema, filling any
user-omitted column with its
+ * default value. Spark 4.2 GA replaced the boolean `supportColDefaultValue`
flag of
+ * `TableOutputResolver.resolveOutputColumns` with a `defaultValueFillMode`
enum, so the call
+ * itself lives in the per-version subclasses.
+ */
+ protected def resolveOutputColumns(tblName: String,
+ expectedColumns: Seq[Attribute],
+ query: LogicalPlan,
+ byName: Boolean): LogicalPlan
+
protected def preprocess(insert: InsertIntoStatement,
catalogTable: Option[CatalogTable]):
InsertIntoStatement = {
preprocess(insert, catalogTable,
catalogTable.map(_.partitionSchema).getOrElse(new StructType()))
@@ -100,13 +112,11 @@ abstract class HoodieSpark4ResolveColumnsForInsertInto
extends ResolveInsertionB
insert.query
}
val newQuery = try {
- TableOutputResolver.resolveOutputColumns(
+ resolveOutputColumns(
tblName,
expectedColumns,
query,
- byName = hasColumnList || insert.byName,
- conf,
- supportColDefaultValue = true)
+ byName = hasColumnList || insert.byName)
} catch {
case e: AnalysisException if staticPartCols.nonEmpty &&
(e.getErrorClass ==
"INSERT_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS" ||
diff --git
a/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark40Analysis.scala
b/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark40Analysis.scala
index 200fe39af471..4949738c44b7 100644
---
a/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark40Analysis.scala
+++
b/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark40Analysis.scala
@@ -21,7 +21,9 @@ import org.apache.hudi.{DefaultSource, EmptyRelation,
HoodieBaseRelation}
import org.apache.hudi.SparkAdapterSupport.sparkAdapter
import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.TableOutputResolver
import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution.datasources.{HadoopFsRelation,
LogicalRelation}
@@ -101,4 +103,11 @@ case class HoodieSpark40ResolveColumnsForInsertInto()
extends HoodieSpark4Resolv
case _ => plan
}
}
+
+ override protected def resolveOutputColumns(tblName: String,
+ expectedColumns: Seq[Attribute],
+ query: LogicalPlan,
+ byName: Boolean): LogicalPlan =
+ TableOutputResolver.resolveOutputColumns(
+ tblName, expectedColumns, query, byName, conf, supportColDefaultValue =
true)
}
diff --git
a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark41Analysis.scala
b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark41Analysis.scala
index 1a57c2976719..7ac107ac341b 100644
---
a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark41Analysis.scala
+++
b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark41Analysis.scala
@@ -21,7 +21,9 @@ import org.apache.hudi.{DefaultSource, EmptyRelation,
HoodieBaseRelation}
import org.apache.hudi.SparkAdapterSupport.sparkAdapter
import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.TableOutputResolver
import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution.datasources.{HadoopFsRelation,
LogicalRelation}
@@ -101,4 +103,11 @@ case class HoodieSpark41ResolveColumnsForInsertInto()
extends HoodieSpark4Resolv
case _ => plan
}
}
+
+ override protected def resolveOutputColumns(tblName: String,
+ expectedColumns: Seq[Attribute],
+ query: LogicalPlan,
+ byName: Boolean): LogicalPlan =
+ TableOutputResolver.resolveOutputColumns(
+ tblName, expectedColumns, query, byName, conf, supportColDefaultValue =
true)
}
diff --git
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/Spark42HoodiePartitionValues.scala
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/Spark42HoodiePartitionValues.scala
index aa53fa4a5b58..91229b2720d0 100644
---
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/Spark42HoodiePartitionValues.scala
+++
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/Spark42HoodiePartitionValues.scala
@@ -20,7 +20,7 @@
package org.apache.hudi
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.unsafe.types.{GeographyVal, GeometryVal}
+import org.apache.spark.unsafe.types.BinaryView
case class Spark42HoodiePartitionValues(override val values: InternalRow)
extends Spark4HoodiePartitionValues(values) {
@@ -29,11 +29,7 @@ case class Spark42HoodiePartitionValues(override val values:
InternalRow)
Spark42HoodiePartitionValues(values.copy())
}
- override def getGeography(ordinal: Int): GeographyVal = {
- values.getGeography(ordinal)
- }
-
- override def getGeometry(ordinal: Int): GeometryVal = {
- values.getGeometry(ordinal)
+ override def getBinaryView(ordinal: Int): BinaryView = {
+ values.getBinaryView(ordinal)
}
}
diff --git
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/client/model/Spark42HoodieInternalRow.scala
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/client/model/Spark42HoodieInternalRow.scala
index 74a5355404b4..bfcfc5a3ee42 100644
---
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/client/model/Spark42HoodieInternalRow.scala
+++
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/hudi/client/model/Spark42HoodieInternalRow.scala
@@ -19,7 +19,7 @@
package org.apache.hudi.client.model
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.unsafe.types.{GeographyVal, GeometryVal, UTF8String}
+import org.apache.spark.unsafe.types.{BinaryView, UTF8String}
class Spark42HoodieInternalRow(
metaFields: Array[UTF8String],
@@ -27,14 +27,9 @@ class Spark42HoodieInternalRow(
sourceContainsMetaFields: Boolean)
extends Spark4HoodieInternalRow(metaFields, sourceRow,
sourceContainsMetaFields) {
- override def getGeography(ordinal: Int): GeographyVal = {
- ruleOutMetaFieldsAccess(ordinal, classOf[GeographyVal])
- sourceRow.getGeography(rebaseOrdinal(ordinal))
- }
-
- override def getGeometry(ordinal: Int): GeometryVal = {
- ruleOutMetaFieldsAccess(ordinal, classOf[GeometryVal])
- sourceRow.getGeometry(rebaseOrdinal(ordinal))
+ override def getBinaryView(ordinal: Int): BinaryView = {
+ ruleOutMetaFieldsAccess(ordinal, classOf[BinaryView])
+ sourceRow.getBinaryView(rebaseOrdinal(ordinal))
}
override protected def newInternalRow(metaFields: Array[UTF8String],
diff --git
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark42Analysis.scala
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark42Analysis.scala
index 7d35378aa000..72d0911d31f8 100644
---
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark42Analysis.scala
+++
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark42Analysis.scala
@@ -21,7 +21,9 @@ import org.apache.hudi.{DefaultSource, EmptyRelation,
HoodieBaseRelation}
import org.apache.hudi.SparkAdapterSupport.sparkAdapter
import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.analysis.TableOutputResolver
import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution.datasources.{HadoopFsRelation,
LogicalRelation}
@@ -101,4 +103,12 @@ case class HoodieSpark42ResolveColumnsForInsertInto()
extends HoodieSpark4Resolv
case _ => plan
}
}
+
+ override protected def resolveOutputColumns(tblName: String,
+ expectedColumns: Seq[Attribute],
+ query: LogicalPlan,
+ byName: Boolean): LogicalPlan =
+ TableOutputResolver.resolveOutputColumns(
+ tblName, expectedColumns, query, byName, conf,
+ defaultValueFillMode = TableOutputResolver.DefaultValueFillMode.FILL)
}
diff --git
a/packaging/bundle-validation/base/build_flink1200hive313spark420previewscala213.sh
b/packaging/bundle-validation/base/build_flink1200hive313spark420scala213.sh
similarity index 80%
rename from
packaging/bundle-validation/base/build_flink1200hive313spark420previewscala213.sh
rename to
packaging/bundle-validation/base/build_flink1200hive313spark420scala213.sh
index 56fad3594a70..bcd6456fc575 100755
---
a/packaging/bundle-validation/base/build_flink1200hive313spark420previewscala213.sh
+++ b/packaging/bundle-validation/base/build_flink1200hive313spark420scala213.sh
@@ -20,10 +20,10 @@
docker build \
--build-arg HIVE_VERSION=3.1.3 \
--build-arg FLINK_VERSION=1.20.1 \
- --build-arg SPARK_VERSION=4.2.0-preview4 \
+ --build-arg SPARK_VERSION=4.2.0 \
--build-arg SPARK_HADOOP_VERSION=3 \
- --build-arg HADOOP_VERSION=3.4.3 \
+ --build-arg HADOOP_VERSION=3.5.0 \
--build-arg SCALA_VERSION=2.13 \
--build-arg DERBY_VERSION=10.16.1.1 \
- -t hudi-ci-bundle-validation-base:flink1200hive313spark420previewscala213 .
-docker image tag
hudi-ci-bundle-validation-base:flink1200hive313spark420previewscala213
apachehudi/hudi-ci-bundle-validation-base:flink1200hive313spark420previewscala213
+ -t hudi-ci-bundle-validation-base:flink1200hive313spark420scala213 .
+docker image tag
hudi-ci-bundle-validation-base:flink1200hive313spark420scala213
apachehudi/hudi-ci-bundle-validation-base:flink1200hive313spark420scala213
diff --git a/packaging/bundle-validation/ci_run.sh
b/packaging/bundle-validation/ci_run.sh
index 64cbf0d53caf..178b647dd033 100755
--- a/packaging/bundle-validation/ci_run.sh
+++ b/packaging/bundle-validation/ci_run.sh
@@ -147,16 +147,16 @@ elif [[ ${SPARK_RUNTIME} == 'spark4.1.1' &&
${SCALA_PROFILE} == 'scala-2.13' ]];
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
IMAGE_TAG=flink1200hive313spark411scala213
-elif [[ ${SPARK_RUNTIME} == 'spark4.2.0-preview4' && ${SCALA_PROFILE} ==
'scala-2.13' ]]; then
- HADOOP_VERSION=3.4.3
+elif [[ ${SPARK_RUNTIME} == 'spark4.2.0' && ${SCALA_PROFILE} == 'scala-2.13'
]]; then
+ HADOOP_VERSION=3.5.0
HIVE_VERSION=3.1.3
DERBY_VERSION=10.14.1.0
FLINK_VERSION=1.20.1
- SPARK_VERSION=4.2.0-preview4
+ SPARK_VERSION=4.2.0
SPARK_HADOOP_VERSION=3
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
- IMAGE_TAG=flink1200hive313spark420previewscala213
+ IMAGE_TAG=flink1200hive313spark420scala213
fi
# Copy bundle jars to temp dir for mounting
diff --git a/packaging/bundle-validation/run_docker_java17.sh
b/packaging/bundle-validation/run_docker_java17.sh
index 8164c0b3092e..a34ae1f77d74 100755
--- a/packaging/bundle-validation/run_docker_java17.sh
+++ b/packaging/bundle-validation/run_docker_java17.sh
@@ -103,16 +103,16 @@ elif [[ ${SPARK_RUNTIME} == 'spark4.1.1' &&
${SCALA_PROFILE} == 'scala-2.13' ]];
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
IMAGE_TAG=flink1200hive313spark411scala213
-elif [[ ${SPARK_RUNTIME} == 'spark4.2.0-preview4' && ${SCALA_PROFILE} ==
'scala-2.13' ]]; then
- HADOOP_VERSION=3.4.3
+elif [[ ${SPARK_RUNTIME} == 'spark4.2.0' && ${SCALA_PROFILE} == 'scala-2.13'
]]; then
+ HADOOP_VERSION=3.5.0
HIVE_VERSION=3.1.3
DERBY_VERSION=10.14.1.0
FLINK_VERSION=1.20.1
- SPARK_VERSION=4.2.0-preview4
+ SPARK_VERSION=4.2.0
SPARK_HADOOP_VERSION=3
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
- IMAGE_TAG=flink1200hive313spark420previewscala213
+ IMAGE_TAG=flink1200hive313spark420scala213
fi
# build docker image
diff --git a/packaging/bundle-validation/validate.sh
b/packaging/bundle-validation/validate.sh
index a432d12c6272..f2b05325e567 100755
--- a/packaging/bundle-validation/validate.sh
+++ b/packaging/bundle-validation/validate.sh
@@ -80,7 +80,7 @@ use_default_java_runtime () {
test_spark_hadoop_mr_bundles () {
echo "::warning::validate.sh setting up hive metastore for spark &
hadoop-mr bundles validation"
- if [ "$SPARK_VERSION" = "4.0.0" ] || [ "$SPARK_VERSION" = "4.1.1" ] || [
"$SPARK_VERSION" = "4.2.0-preview4" ]; then
+ if [[ "$SPARK_VERSION" == 4.* ]]; then
change_java_runtime_version
fi
$DERBY_HOME/bin/startNetworkServer -h 0.0.0.0 &
@@ -112,7 +112,7 @@ test_spark_hadoop_mr_bundles () {
numRecordsHiveQL=$(cat $hiveqlresultsdir/*.csv | wc -l)
if [ "$numRecordsHiveQL" -ne 10 ]; then
echo "::error::validate.sh HiveQL validation failed."
- if [ "$SPARK_VERSION" = "4.0.0" ] || [ "$SPARK_VERSION" = "4.1.1" ] ||
[ "$SPARK_VERSION" = "4.2.0-preview4" ]; then
+ if [[ "$SPARK_VERSION" == 4.* ]]; then
echo "::error::validate.sh Debug info for Spark4 validation
failure:"
$HIVE_HOME/bin/beeline --hiveconf
hive.input.format=org.apache.hudi.hadoop.HoodieParquetInputFormat \
-u jdbc:hive2://localhost:10000/default
--showHeader=true --outputformat=csv2 \
@@ -259,7 +259,7 @@ test_metaserver_bundle () {
java -jar $JARS_DIR/metaserver.jar &
local METASEVER_PID=$!
- if [ "$SPARK_VERSION" = "4.0.0" ] || [ "$SPARK_VERSION" = "4.1.1" ] || [
"$SPARK_VERSION" = "4.2.0-preview4" ]; then
+ if [[ "$SPARK_VERSION" == 4.* ]]; then
change_java_runtime_version
fi
echo "::warning::validate.sh Start hive server"
diff --git a/pom.xml b/pom.xml
index a5a4d0685cd5..c7c66e712c73 100644
--- a/pom.xml
+++ b/pom.xml
@@ -178,7 +178,7 @@
<spark35.version>3.5.5</spark35.version>
<spark40.version>4.0.2</spark40.version>
<spark41.version>4.1.1</spark41.version>
- <spark42.version>4.2.0-preview4</spark42.version>
+ <spark42.version>4.2.0</spark42.version>
<hudi.spark.module>hudi-spark3.5.x</hudi.spark.module>
<hudi.spark.common.module>hudi-spark3-common</hudi.spark.common.module>
<avro.version>1.11.4</avro.version>
@@ -3002,7 +3002,7 @@
<!-- This glob has to include hudi-spark4-common -->
<hudi.spark.common.module>hudi-spark4-common</hudi.spark.common.module>
<scalatest.version>${scalatest.spark4.version}</scalatest.version>
- <hadoop.version>3.4.3</hadoop.version>
+ <hadoop.version>3.5.0</hadoop.version>
<kafka.version>3.9.2</kafka.version>
<hive.storage.version>2.8.1</hive.storage.version>
<!-- TODO: Enable lance tests on Spark 4.2 -->
@@ -3031,7 +3031,7 @@
both provide the same net.jpountz.lz4 package so we must use the
Spark-provided one
to avoid classpath conflicts (the newer version adds
LZ4BlockInputStream.newBuilder()) -->
<lz4.groupId>at.yawk.lz4</lz4.groupId>
- <lz4.version>1.10.4</lz4.version>
+ <lz4.version>1.11.0</lz4.version>
<skip.hudi-spark2.unit.tests>true</skip.hudi-spark2.unit.tests>
<skipITs>true</skipITs>
</properties>
