andygrove commented on PR #6664: URL: https://github.com/apache/datafusion-comet/pull/6664#issuecomment-6024238882
Thanks @comphead. Writing out the four combinations showed we didn't need two settings: the native flag did nothing without the split flag, and the split plan without the native writer gives users nothing over Spark's own operator. So instead of documenting the matrix, I've made `spark.comet.write.iceberg.enabled` the only setting. On, which is the default, Comet plans `IcebergCommit` over `IcebergWrite` and writes eligible data files natively, with iceberg-java writing the rest inside the same plan. Off, Spark plans its own write operator, as in 1.1.0. `spark.comet.write.iceberg.splitOperator.enabled` is back to being a testing-only setting, which the suites use to get the split plan with the native writer off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
