LuciferYang commented on PR #58527: URL: https://github.com/apache/spark/pull/58527#issuecomment-5548585266
`bestSpecOpt` is still picked with `ShuffleSpecCollection.numPartitions`, which comes from `specs.head`. A collection like [coarse head with 2 partitions, fine member with 200] competes as 2 in the `maxBy`, loses to a sibling child's plain 100-partition spec, and the side holding the fine member gets re-shuffled onto 100 partitions when it could have stayed unshuffled as the best. The selection line predates this PR, and in this shape it behaves exactly as before (the losing side took the same full re-shuffle), so this is purely pre-existing. Not blocking; worth a follow-up comparing best candidates at member level at the call site (changing `numPartitions` itself would affect other readers). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
