peter-toth commented on PR #58339: URL: https://github.com/apache/spark/pull/58339#issuecomment-5552062951
@ulysses-you one note on SPARK-59272, since it grew out of finding 22 here. That direction overlaps with a larger SPJ refactor we have designed but not filed: deciding a co-partitioned child's `GroupPartitionsExec` once, at the site that already holds the join's merged keys, instead of inserting a placeholder and rewriting it afterwards. Committing to the pairing before knowing what the regrouping can promise is one of the things that reordering removes, so SPARK-59272 sits inside it. We said on #58279 round 3 that we were keeping that direction, and we have some measurements for it, including a cheap sufficient condition under the default config. Would you mind if we took SPARK-59272 as part of that work? If you would rather land a narrower guard here first, that is fine too and we would build on it — I would just like to avoid us both writing the same check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
