peter-toth commented on PR #58351: URL: https://github.com/apache/spark/pull/58351#issuecomment-5450905194
Two corrections after a closer look at how this relates to #58316. **Reclassified SPARK-59057 as a Bug** (it was an Improvement), affected versions 4.3.0 / 4.4.0 / 5.0.0. Provenance leaves the gate open in three ways, and the description now leads with them rather than with the false refusals: - The flag is dropped wherever a partitioning is rebuilt, so the gate cannot see a real risk. That is SPARK-59026's defect, fixed here. - A reduction never sets the flag at all: under `allowCompatibleTransforms`, a `bucket(4, id)` side reduced onto a `bucket(2, id)` join maps four distinct keys onto two, but drops no key position, so provenance misses the entire class and the opt-in is bypassed. - A `PartitioningCollection` whose members disagreed could be entered through the plain one, since `EnsureRequirements` uses `find`. **Took the shuffle-template chain from #58316's end-to-end test**, with the expectation these semantics call for. I had left it out because its `items` has unique ids, so dropping `name` collapses nothing and all three of its assertions flip -- but that left the chain uncovered. It is now a positive test for the reclassification: the union's duplicate keys come from the two children holding the same ids rather than from the projection, so the final aggregate groups them with the opt-in off. Verified it fails on the old formula. @dongjoon-hyun your `DistributionSuite` test was already in, adapted to the new flag name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
