peter-toth commented on PR #58659: URL: https://github.com/apache/spark/pull/58659#issuecomment-5679358702
@ulysses-you you are right, the two are better kept apart. Sorry for the muddle: on #58339 I asked to take SPARK-59272 inside this refactor, and your original split was the better call. SPARK-59272 is now its own PR: #58814. It is the pairwise check your ticket describes, 28 lines in `checkKeyGroupCompatible`, and the `SPARK-59050` regrouping test is the only one of `KeyGroupedPartitioningSuite`'s 183 that it moves. On the 307200-plan sweep it accounts for the whole `2024 -> 0` line on its own: the co-partitioning violations it removes are exactly the validation failures it removes, and it adds no case anywhere. I will rebase this PR onto that one, and onto master once it lands. So the release note for the pairing gate hangs off SPARK-59272, and this PR stays SPARK-59289, the refactor. On a backport, since you asked which ticket it would hang off: the gate applies to `branch-4.x` as it stands, because the marker, `alignToExpectedKeys` and `describesSameKeys` are all already there. The older branches would need more, `describesSameKeys` is not on them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
