felipepessoto opened a new issue, #13500: URL: https://github.com/apache/gluten/issues/13500
### Description Add native Velox write support for Delta Lake's [Materialize Partition Columns](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#materialize-partition-columns) writer feature (`materializePartitionColumns`) in the Spark 4.x/Delta integration. Partitioned tables using this feature currently fall back to Delta's JVM writer, preventing native write acceleration even when the remaining write configuration is supported. This is a native-write support gap, not a claim that the tables cannot be read or written through Delta. #### Current behavior - The [Velox Delta documentation](https://github.com/apache/gluten/blob/main/docs/backends/velox/get-started/VeloxDelta.md#native-write-on-spark-4x) lists partition-column materialization among native-write fallback conditions; the [table-feature matrix](https://github.com/apache/gluten/blob/main/docs/backends/velox/get-started/VeloxDelta.md#table-features) marks `materializePartitionColumns` writes as **Fallback**. - [`GlutenOptimisticTransaction`](https://github.com/apache/gluten/blob/3071f767a0f260804c6904bba74f51d9ce9dd94e/backends-velox/src-delta4/main/scala/org/apache/spark/sql/delta/GlutenOptimisticTransaction.scala#L137-L151) already determines whether partition columns must be written from Iceberg compatibility, `MaterializePartitionColumnsTableFeature`, and `delta.writePartitionColumnsToParquet`. - [`GlutenDeltaNativeWriteGate`](https://github.com/apache/gluten/blob/main/backends-velox/src-delta4/main/scala/org/apache/spark/sql/delta/GlutenDeltaNativeWriteGate.scala#L98-L100) rejects partitioned writes requiring materialization with: `Partition columns in data files are not written by the Velox writer`. The documentation also notes that writing partition columns into data files is Delta Lake 4.4's default, so the underlying capability is relevant beyond explicitly enabling this table feature. #### Requested behavior and protocol requirements Implement partition-column materialization in the native write path, then allow eligible writes through the gate while preserving unrelated safety fallbacks. - Recognize writer version 7 with `materializePartitionColumns` in `writerFeatures`. Its presence alone activates the requirement; there is no additional `delta.enable*` property. - Physically write partition columns into every newly added Parquet data file referenced by an `AddFile` action. Each value, including null, must match the corresponding logical partition value in `AddFile.partitionValues`. - Honor Delta's resolved `delta.writePartitionColumnsToParquet` behavior. The writer feature requires materialization even when the property is `false`; without a requiring feature, respect explicit `true`/`false` values and Delta's default when unset. Do not equate that implementation default with a protocol mandate. - For column-mapped tables, materialized columns must use their assigned physical names and field IDs. Keep fallback for independently unsupported combinations until those requirements can be met; this request does not require full UniForm support. - Preserve existing reader semantics: Delta readers obtain partition values from `AddFile` metadata, regardless of their presence in Parquet. Do not emit redundant file-level statistics for partition columns. This writer feature introduces no new reader requirements. #### Acceptance criteria - Otherwise-supported partitioned writes with the feature enabled actually use the native writer, including existing native-write commands that add or rewrite data files; correctness through fallback alone is insufficient. - Add regression coverage for feature present/absent with the table property `true`, `false`, and unset, including feature precedence over `false`, null partition values, and multiple partition columns. - Inspect physical Parquet schemas and values directly, rather than relying on partition discovery, and validate `AddFile.partitionValues` and the absence of partition-column file statistics. Compare behavior with vanilla Delta. - Verify interoperability and unchanged reads for tables containing both older non-materialized files and newly materialized files, retain fallback coverage for unsupported combinations, and update the Velox Delta support documentation. Related: [#12039](https://github.com/apache/gluten/issues/12039) tracks UniForm Iceberg support, which shares the partition-materialization requirement but has additional compatibility requirements. Prepared with assistance from GitHub Copilot. ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
