peter-toth commented on PR #58351: URL: https://github.com/apache/spark/pull/58351#issuecomment-5450531273
Rebased on master and squashed to a single commit, so #58338 is out of the diff now that it has landed. The description is updated to match. Three other things went in with the rebase, each from a review pass over the change: - The config doc for `allowKeysSubsetOfPartitionKeys` was still phrased in the old vocabulary and had become wrong -- it claimed the config gates "a partitioning that was narrowed to a subset of its keys and whose keys are no longer distinct", while under these semantics a subset projection that keeps every distinct key is *not* gated and a reduction that drops no key position *is*. Two of this PR's tests pin both directions. - `GroupPartitionsExec` read the flag from every member of the child's partitioning, which forced `distinctKeyCount` -- a pass over the keys -- once per member for an answer they share. It reads one representative now. - I dropped the unit test that asserted the flag is taken from every member of a `PartitioningCollection`: once the collection normalizes the flag, head and `exists` agree by invariant, so that test could not fail. The normalization test covers the invariant instead, and the description's ablation list now names only the ablations that actually bite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
