malinjawi opened a new pull request, #13324:
URL: https://github.com/apache/gluten/pull/13324
## What changes are proposed in this pull request?
With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite` on,
UPDATE, DELETE, OPTIMIZE and CTAS already write their data files natively, but
MERGE still goes through Delta's writer. This PR adds MERGE.
- `OffloadDeltaCommand` wraps `MergeIntoCommand` in a new
`GlutenDeltaMergeIntoCommand`, which runs it in a Gluten transaction like the
other commands.
- A new flag,
`spark.gluten.sql.columnar.backend.velox.delta.enableNativeMergeWrite` (default
true), turns off just MERGE.
- MERGE keeps Delta's writer for tables the native writer doesn't get right
yet:
- column mapping (native files have no Parquet field ids)
- IcebergCompat
- `materializePartitionColumns`
- FLOAT/DOUBLE partition columns (#13138)
- TIMESTAMP_NTZ
- nested upper-case field names under case-insensitive analysis
- stats collection off
- concurrent output writers
- `maxRecordsPerFile` on unpartitioned tables
- The check runs at planning and again in `writeFiles` against the
transaction's own metadata, in case the table changed in between.
- If another transaction observer is installed on the thread, MERGE keeps
Delta's writer so that observer isn't replaced. This can go once #13134 lands.
- The Spark 3.5 `GlutenOptimisticTransaction` gets the Velox schema fallback
that the Spark 4.x one already has.
It's a draft because the Delta test harness turns native write on, so its
MERGE suites now write natively. I'll go through what that run shows before
marking this ready.
## How was this patch tested?
- `GlutenDeltaMergeWriteGateSuite` has one case per fallback reason. It
passes locally on Spark 3.5 and 4.1.
- `GlutenDeltaNativeMergeWriteSuite` (gluten-ut):
- Update/delete/insert, insert-only, partitioned, CDF, deletion-vector and
DeltaTable API MERGEs write native files and read back the same as a twin table
written without Gluten.
- Each fallback case, and the observer case, keeps Delta's writer.
- The Delta UT gate runs on this PR.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
