andygrove opened a new issue, #5644: URL: https://github.com/apache/datafusion-comet/issues/5644
### What is the problem the feature request solves? Native Iceberg writes ship behind two flags that both default to `false`: `spark.comet.write.iceberg.splitOperator.enabled` (the writer/committer split from #4658) and `spark.comet.iceberg.write.enabled` (the iceberg-rust data-file writer from #5361). Nothing in CI runs with either flag on, so the feature is only exercised by the suites that set the flags themselves, and there is no agreed definition of what has to be true before the defaults flip. #5259 catalogued what breaks when the split plan is enabled by default (8 jobs, 4 buckets). That is the first step, but it does not cover the native writer flag or say what else is required. ### Describe the potential solution Graduation criteria, each of which should be a linked issue or a checked item before the defaults change: - [ ] #5259: the four failure buckets when the split plan is on by default are fixed - [ ] A CI job in `pr_build_linux.yml` runs the Iceberg suites with both flags on (or the suites default them on), so a regression on the native path fails a PR rather than a manual run - [ ] The two correctness bugs on the native path are fixed: #5636 (partition-path escaping) and #5637 (GCS configuration) - [ ] Failure handling matches iceberg-java: #5618 (task-attempt cleanup) and #5277 (orphans on commit failure), with failure-injection tests - [ ] Manifest metrics parity is asserted per data type across all four Iceberg versions in the build matrix (1.5.2, 1.8.1, 1.10.0, 1.11.0), not only for the hand-picked cases in `CometIcebergWriteActionSuite` - [ ] The default hash-distribution write is native end to end (#5635) - [ ] The Spark SQL / Iceberg test diffs under `dev/diffs` pass with the flags on - [ ] A write benchmark exists and shows no regression against iceberg-java for unpartitioned, clustered, and fanout writes - [ ] The eligibility restrictions each have a keep-or-lift decision, so the fallback surface is intentional Suggested rollout: flip `splitOperator.enabled` first (it changes plan shape but still writes through iceberg-java, so the blast radius is planning only), then `iceberg.write.enabled` one release later. ### Additional context Part of the native Iceberg writes epic. Related: #4658, #5298, #5361. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
