andygrove opened a new issue, #6142:
URL: https://github.com/apache/datafusion-comet/issues/6142
### Describe the bug
`IcebergWriteStrategy` checks only
`spark.comet.write.iceberg.splitOperator.enabled`
(`spark/src/main/scala/org/apache/comet/iceberg/IcebergWriteStrategy.scala:38`),
and the strategy is registered unconditionally
(`CometSparkSessionExtensions.scala:102`). It does not check
`spark.comet.enabled` or whether Comet is loaded.
Today the split flag defaults to `false`, so this has no effect. Once it
defaults to `true` (#5644), a session with `spark.comet.enabled=false` would
still plan Iceberg writes as Comet's `IcebergWriteExec` + `IcebergCommitExec`.
The main kill switch would no longer restore Spark's own write plan, which is
the first thing users try when they suspect Comet.
### Steps to reproduce
```scala
spark.conf.set("spark.comet.enabled", "false")
spark.conf.set("spark.comet.write.iceberg.splitOperator.enabled", "true")
spark.sql("INSERT INTO iceberg_table SELECT ...").explain()
// plan contains IcebergCommitExec / IcebergWriteExec
```
### Expected behavior
With `spark.comet.enabled=false` (or Comet not loaded), the strategy returns
`Nil` and Spark plans its own `AppendDataExec` / `ReplaceDataExec` and so on.
Gate it the same way `CometRule` gates conversion.
### Additional context
This blocks flipping the split-operator default, the first step of the
rollout in #5644. Found in an audit of the native Iceberg write path. Part of
#5649.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
