malinjawi opened a new pull request, #12957: URL: https://github.com/apache/gluten/pull/12957
### What changes were proposed in this pull request? A Delta table with IcebergCompatV2 is supposed to produce parquet files whose footers carry Iceberg-compatible field ids. Vanilla Delta arranges that in `prepareWrite` by installing `DeltaParquetWriteSupport`, but that only configures parquet-mr — when the write is offloaded, Velox writes the file directly and never consults it. The ids can't leak in through Arrow either, since `SparkArrowUtil.toArrowField` drops `StructField` metadata. So a UniForm table written natively ends up with no field ids at all, with no error and no fallback, while the `IcebergConverter` happily publishes Iceberg metadata over those files. This reads the ids Delta already recorded in the schema — `parquet.field.id` for columns, `parquet.field.nested.ids` for the synthetic list-element / map-key / map-value nodes — and passes them through the write options into `ParquetWriterOptions`. Gated on `IcebergCompatV2`, so nothing changes for other tables. Mirrored into `src-delta40`, where the helper is identical and symlinked. Native Delta write is still experimental and off by default, so this closes a correctness hole ahead of that flag rather than fixing shipped behaviour. ### How was this patch tested? `GlutenDeltaParquetFieldIdSuite` covers the encoding. `DeltaUniFormIcebergSuite` writes a partitioned table with nested struct/array/map columns, checks the footer with Delta's own `isParquetIcebergCompatV2`, and reads the table back through Iceberg. ### Notes Draft, and there's a known gap I'd rather flag than hide: the end-to-end suite drives the write with `INSERT INTO`, which `OffloadDeltaCommand` doesn't offload, so it currently exercises the vanilla path and would pass with this change reverted. I'm reworking it onto an offloaded write with an explicit offload assertion before this is ready for review. Two open questions I'd welcome opinions on: - The gate could reasonably be "the schema carries `parquet.field.id`" rather than `IcebergCompatV2`. Delta enables field-id write for every session and writes the key for both `name` and `id` mapping, so the native writer is dropping ids for ordinary column-mapped tables too — the wider gate would cover IcebergCompatV1 and id mapping in the same guard. - Missing metadata currently yields `-1`. Delta's own `getNestedFieldId` throws instead, and Velox omits the annotation entirely for negative ids, so failing fast is probably the better behaviour. This is the write-side counterpart to #12884, which plumbs column mapping mode through scan splits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
