dwsmith1983 commented on PR #6785: URL: https://github.com/apache/datafusion-comet/pull/6785#issuecomment-6076350015
@andygrove I'd suggest this for `backport-1.1`. `branch-1.1` has the same `RewriteJoin`, so with `spark.comet.exec.forceShuffledHashJoin` on, 1.1.0 drops the same sorts: `sortWithinPartitions` and `SORT BY` on the join key return unsorted partitions, and a partitioned insert keyed on the join key fails with `FileAlreadyExistsException`. The branch has no #6384 size limit, so the backport would bring the strategy over without the size check rather than cherry-pick this as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
