andygrove opened a new pull request, #6407: URL: https://github.com/apache/datafusion-comet/pull/6407
## Which issue does this PR close? No issue; documentation only. Follow-up to review feedback from @mbutrovich on the Comet 1.1.0 blog post (apache/datafusion-site#205), which repeated this wording. ## Rationale for this change Both Iceberg writes guides say the query feeding an Iceberg write can't be re-planned by AQE, and that the split-operator plan makes the write's input visible to AQE. That isn't right. Spark's `InsertAdaptiveSparkPlan` has `case c: V2CommandExec => c.withNewChildren(c.children.map(apply))`, in both Spark 3.5 and 4.0, so the input query of a V2 write such as `AppendDataExec` is already wrapped in AQE. What sits outside AQE is the write operator itself. As #4658 describes it, the split moves the data-file writing step inside AQE, so it can be re-planned in response to its upstream operators, and separates it from the metadata writing and commit, which Comet does not accelerate. ## What changes are included in this PR? - `user-guide/latest/iceberg-writes.md`: the Overview now says AQE already re-plans the sub-query, that the write operator is what sits outside AQE, and that the split moves data-file writing inside AQE, apart from the commit. - `contributor-guide/iceberg-writes.md`: the Split-Operator Plan section says the same and names `InsertAdaptiveSparkPlan`. The plan diagram now marks which operators run inside and outside AQE instead of labeling the input query "now visible to AQE and Comet". ## How are these changes tested? Documentation only. `npx prettier@latest --check` passes on both files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
