Shekharrajak opened a new pull request, #3524: URL: https://github.com/apache/datafusion-comet/pull/3524
## Which issue does this PR close? Closes #3429 ## Rationale for this change CTAS with UNION fails in Spark 4.x when the Comet native writer is enabled. The `requiresNativeChildren` check in `CometExecRule` was too strict - it required children to be `CometNativeExec`, but sink operators like `CometUnionExec`, `CometCoalesceExec`, `CometCollectLimitExec` extend `CometExec` directly while still producing Arrow-formatted data. ## What changes are included in this PR? - Changed the type check from `CometNativeExec` to `CometExec` in `CometExecRule.convertToComet()` to allow all Comet operators that produce Arrow-formatted data as input to the native writer. ## How are these changes tested? Added 6 new test cases in `CometParquetWriterSuite`: - Basic UNION write (CTAS style) - Multiple (3+) DataFrames UNION -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
