andygrove opened a new pull request, #5677: URL: https://github.com/apache/datafusion-comet/pull/5677
## Which issue does this PR close? None — exploratory. Related to #5308 (native Iceberg write) and the split-operator work in #4658. **Draft, not for merge.** The point is to find out what breaks. ## Rationale for this change `spark.comet.write.iceberg.splitOperator.enabled` and `spark.comet.iceberg.write.enabled` have both defaulted to `false` since they landed. That means no CI job has ever run the two-operator plan or the iceberg-rust writer except the handful of Comet suites that set them explicitly — `CometIcebergWriteActionSuite`, `CometIcebergWriteDetectionSuite`, `CometIcebergRewriteActionSuite`, `CometIcebergSystemFunctionSuite`. Everything else, including the Iceberg Spark SQL suites that run Iceberg's own tests against Comet, has been exercising Spark's stock combined write operator. So we do not currently know how much of the Iceberg test surface the split plan and the native writer actually pass. This flips both defaults to find out. ## What changes are included in this PR? Two lines in `CometConf.scala`, plus dropping the now-false "Off by default" sentence from the native-write config doc. Deliberately **not** included: `docs/source/user-guide/latest/iceberg-writes.md` still says both features are experimental and off by default, and still lists `splitOperator.enabled` being `false` as the first fallback reason. If any of this is worth keeping, the guide has to be rewritten — but that is premature until we see the failures. ## How are these changes tested? By CI, which is the whole point. The `run-iceberg-tests` label is applied so the Iceberg 1.8 / 1.9 / 1.10 jobs run alongside 1.11, giving coverage across Spark 3.4 / 3.5 / 4.1 and four Iceberg versions. Comet's own Iceberg suites run in the `[scans]` bucket on every profile regardless. Nothing has been run locally beyond a compile — the intent is to read CI rather than to pre-filter it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
