andygrove opened a new issue, #5649: URL: https://github.com/apache/datafusion-comet/issues/5649
### What is the problem the feature request solves? Native Iceberg writes landed in three parts under #4322: the split writer/committer plan (#4658), the eligibility allowlist with fall-back reasons (#5298), and the iceberg-rust data-file writer with JVM-side manifest metrics (#5361). #4322 closed when #5361 merged, and no issue tracks what remains between "merged behind two off-by-default flags" and "production quality". The open follow-ups exist but are not linked to each other, and several gaps had no issue at all. This epic is that plan. The goal is unchanged from #4322: an ETL job that scans, transforms, and writes Iceberg tables runs natively end to end, and falls back to iceberg-java wherever the native path cannot reproduce its outcome. **Current state.** `spark.comet.write.iceberg.splitOperator.enabled` and `spark.comet.iceberg.write.enabled` both default to `false`. Nothing in CI runs with either flag on. Two correctness bugs are open on the native path. The user guide (`iceberg-writes.md`) still describes the writer as unimplemented in one section. ### Describe the potential solution Phased, in priority order. Each item is a linked issue; check it here when it closes. **Phase 1: correctness blockers** - [ ] #5636 Bump the iceberg-rust pin past the partition-path escaping fix (a `#` in an S3 partition value commits a location iceberg-java resolves to a different key) - [ ] #5637 Native writes drop Hadoop GCS configuration for `HadoopFileIO` + `gs://`; fail closed until the translation is complete - [ ] #5641 Fix the stale "not yet wired in" wording and the divergence list in `iceberg-writes.md` **Phase 2: failure handling parity with iceberg-java** - [ ] #5618 Clean up task-attempt data files on task failure - [ ] #5277 Remove orphaned files when the commit fails - [ ] #5646 Failure-injection tests that assert both the visible outcome and the storage state **Phase 3: coverage so common workloads stay native** - [ ] #5635 Iceberg system functions (`bucket`, `truncate`, `years`/`months`/`days`/`hours`) as native expressions, so the default hash-distribution write is native end to end (PR #5638) - [ ] #5318 Native `MergeRowsExec` - [ ] #5122 Row-level MERGE / UPDATE / DELETE plans - [ ] #5642 Lift the metrics-mode `counts` / `none` restriction now that metrics are assembled on the JVM - [ ] #5643 A keep-or-lift decision for every remaining eligibility restriction, so the fallback surface is intentional - [ ] #5306 Reconcile the two config namespaces for native writes **Phase 4: enable by default** - [ ] #5259 Fix the four CI failure buckets when the split plan is on by default - [ ] #5644 Graduation criteria and the two-step default flip (split plan first, native writer one release later), including a CI job that runs the Iceberg suites with both flags on **Phase 5: performance** - [ ] #5647 A native Iceberg write benchmark in the repository (unpartitioned, default hash distribution, fanout, copy-on-write delete) - [ ] #5648 Account the writer's buffers in Comet's memory pool, especially the fanout writer's per-partition buffers - [ ] #3595 Fused sort plus write for the clustered path **Ongoing** - [ ] #5645 A pin-bump policy for iceberg-rust, so upstream fixes to file bytes, manifests, or partition layout reach Comet on a schedule - [ ] #3756 Update the Iceberg feature matrix, which still lists the write path as not implemented ### Additional context Done and merged: #4658 (split-operator plan), #5298 (eligibility detection), #5361 (native writer). Related umbrellas: #5121 (DataSource V2 writes in general), #1625 (native Parquet writes for the V1 path). The accepted divergences between iceberg-rust and iceberg-java output are documented in `iceberg-writes.md`; changes to that list belong in the issues above rather than here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
