ulysses-you commented on PR #58339: URL: https://github.com/apache/spark/pull/58339#issuecomment-5504387411
Thanks for the thorough check, @dongjoon-hyun -- all three points addressed in 7464c25: 1. Adopted as proposed: the marker is cleared at construction in `ShuffledJoin`'s `InnerLike` arm, membership via `PartitioningCollection.flatten` so nested collections are reached (a hand-built marked-inside-nested unit test pins it, since planned SQL cannot leave a marked member nested), and the consumer-side `forall` scoping in `PartitioningPreservingUnaryExecNode` and `GroupPartitionsExec` is deleted. Your repro became the both-member-orders test; measured without the clearing, exactly the flagged-first arm regresses to 2 shuffles (master: 1 + the GPE pair), and with it both orders match master. 2. Reworded as suggested -- the comment cites the empty key-position intersection and `canCreatePartitioning`, not arity. 3. The `areKeysCompatible: incompatibility without unknown partition keys` test (pre-existing generic behavior I had added in round 1) is dropped; the `toSet` on the marked path is built once per `areKeysCompatible` call, not per key, so it stays as is. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
