malinjawi opened a new pull request, #13271:
URL: https://github.com/apache/gluten/pull/13271
## What changes are proposed in this pull request?
The `spark-4.2` profile still used the Spark 4.1 Delta jar
(`delta-spark_4.1` 4.1.0), with a note that Delta had no Spark 4.2 release.
Delta 4.4.0 (2026-08-20) and 4.4.1 publish `delta-spark_4.2_2.13`, built
against Spark 4.2.0. This PR switches to it and turns on Delta in the Spark 4.2
CI lane.
- **Build:** `pom.xml` and `tools/gluten-it/pom.xml` now use
`delta-spark_4.2` 4.4.1 for `spark-4.2`. Its APIs still fit the `src-delta40`
trees, so `delta.binary.version` stays 40.
- **Tests that no longer compile:** two unused helpers copied from Delta's
tests don't compile against 4.4 and are removed:
`getCheckpointDfForFilesContainingFileActions` and
`TableCommitCoordinatorClientTestHelper.commit`.
- **Partition columns in data files:** Gluten's native partitioned write
always leaves partition columns out of the data files.
- Some tables need them in the files: Iceberg-compat tables, and tables
with the `materializePartitionColumns` writer feature (Delta 4.1+). Those now
fall back to the vanilla Delta write.
- Gluten ignored the feature before this PR, so Spark 4.1 was already
affected. Delta 4.0 has no class for the feature, so the check matches it by
name.
- **Delta 4.4's new default:** Delta 4.4 also writes partition columns into
the files when the feature is absent (`delta.writePartitionColumnsToParquet`,
default true). Those writes stay native and leave the columns out.
- Delta reads partition values from the log, so results are the same.
- This is documented in `VeloxDelta.md`. Native support is a follow-up.
- **Deletion vectors without an offset:** Delta 4.4 allows on-disk deletion
vectors with no offset (delta-io/delta#6436) and reads them from the start of
the file. Gluten threw on them. It now uses offset 0, as Delta's
`DeletionVectorStoredBitmap` does.
- **CI:** `-Pdelta` is added to `build-test-spark42` and
`spark-test-spark42`, matching the Spark 4.1 lane.
- **Docs:** `VeloxDelta.md` gains Spark 4.1 and 4.2 rows.
Running Delta's own test suite on Spark 4.2 (`delta_spark_ut.yml`) is left
to a follow-up. It needs v4.4.1 test patches and a separate known-failures
baseline.
Part of #12569.
## How was this patch tested?
- **Compile:** `test-compile` with `-Pdelta -Pspark-ut` passes on Spark 3.5,
4.0, 4.1 and 4.2.
- **`DeltaDeletionVectorScanInfoSuite`:** a new test points an offset-less
descriptor at a copy of a deletion vector that starts at byte 0. It checks the
payload matches the original.
- Locally, the suite passes on Spark 4.2 (10/10).
- With the fix reverted, the new test fails with the old "missing its
offset" error.
- It is skipped below Delta 4.4.
- **`GlutenDeltaNativePartitionWriteSuite`:** a new test writes to a
partitioned table with `materializePartitionColumns` enabled. It checks that
every data file's Parquet footer contains the partition column. It runs on the
Spark 4.1 and 4.2 lanes and is skipped below Delta 4.1.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
