peter-toth commented on PR #58316: URL: https://github.com/apache/spark/pull/58316#issuecomment-5442460805
The follow-up I promised is open: #58351 (SPARK-59057). It makes the flag mean actual key collapse -- a projection that mapped keys which were distinct in the input onto the same projected key -- instead of "positions were dropped", and renames it to `isCollapsed`. Your finding is in it: all five producers that rebuild a `KeyedPartitioning` now propagate the flag, and your `DistributionSuite` test is taken with credit. Your end-to-end test is not, and that is worth flagging: its `items` table has unique ids, so dropping `name` keeps every key distinct, which under the new contract is not a coarsening at all. That scenario is exactly the one the change reclassifies, so the two PRs cannot both land as written. #58351 says it supersedes this one, but closing this is your call. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
