jordepic opened a new issue, #5308: URL: https://github.com/apache/datafusion-comet/issues/5308
## What is the problem the feature request solves? This is the third and final part of #4322 (accelerating Iceberg V2 write operations), which was split into three deliverables: 1. **Split-operator plan** — #4658 (merged). Iceberg's single V2 write command is rewritten into a writer operator (inside AQE) and a committer operator, with file writing still running through iceberg-java. 2. **Native-write eligibility detection** — #5298. An allowlist over the write's effective configuration decides whether a native write would produce the same outcome as iceberg-java, and reports a fall-back reason via extended EXPLAIN when it would not. The supported configuration and the accepted divergences are documented in `iceberg-writes.md`. This issue tracks part 3: actually performing the per-task Parquet data-file write via [iceberg-rust](https://github.com/apache/iceberg-rust) when detection accepts the write. ## Describe the potential solution High-level approach: - **Conversion**: `CometIcebergNativeWrite.convert` (currently a stub) produces a native write operator. The JVM resolves all write settings and passes them to the native side via protobuf; the vetted `write.parquet.*` configuration (compression codec/level, row-group/page/dictionary sizes, page row limit) is translated into parquet-rs `WriterProperties`. Footer statistics are written untruncated to match parquet-mr. - **Writer modes**: unpartitioned, clustered, and fanout writers, mirroring iceberg-java's writer selection. - **Transport**: the native side serialises the resulting `DataFile`s as an in-memory Iceberg manifest via iceberg-rust's `ManifestWriter`; the JVM decodes the bytes with Iceberg's own `ManifestFiles.read`. Both ends use their library's spec codec, so no hand-rolled Avro. - **Commit**: unchanged from part 1 — the recovered `DataFile[]` goes into `SparkWrite$TaskCommit` and `BatchWrite.commit`, preserving Iceberg's overwrite semantics, copy-on-write validation, WAP, and snapshot summaries. - **Manifest metrics**: assembled on the JVM at commit time using Iceberg's own `MetricsConfig` / `ParquetUtil` code against the written file's footer (plus NaN counts from the native writer, which the footer cannot provide). Java makes every metadata decision — metrics modes, the inferred-column cap, bound truncation, list/map bounds suppression — so the `counts` / `none` metrics restrictions from part 2 can be lifted. - **Testing**: parity tests comparing native output against iceberg-java per data type (including a targeted float/double NaN metrics test), plus end-to-end tests showing compaction / sort / z-order rewrites running fully native. The guiding rule from part 2 carries over: if the native write cannot reproduce iceberg-java's outcome for the effective configuration, the write falls back — the documented accepted divergences (row-group/file-rolling cadence, compressed page bytes, footer key-value metadata, partition-path escaping) are the only exceptions. ## Additional context Detection (#5298) is deliberately conservative so this part can land incrementally: individual restrictions can be relaxed as the native writer proves each behaviour byte- or semantics-identical. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
