ulysses-you commented on PR #58279: URL: https://github.com/apache/spark/pull/58279#issuecomment-5534889309
Thanks @peter-toth -- 9, 10 and 11 are addressed in 94afe6e4401: * **9** -- one descent per side: `leftGrouping`/`rightGrouping` are computed once, and the unwrapped plans, the statistics, the original keys and the positions all derive from them. `unwrapGroupPartitions` keeps its one remaining caller at the shuffle site. * **10** -- the fallback compares the pre-alignment split counts, read through a new `PartitioningCollection.numKeyedPartitions` (the representative keyed member, collections included). A unit test pins it by forcing the fallback -- dummy plans carry no `logicalLink` -- and reads the choice back off the `distributePartitions` flags; reverting the fix fails its first arm (`(true, false)` against the expected `(false, true)`). * **11** -- the comment is in verbatim. Follow-up for finding 5 filed as [SPARK-59234](https://issues.apache.org/jira/browse/SPARK-59234); finding 2 is left with you as discussed. Local suites green (204). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
