ulysses-you commented on code in PR #58339: URL: https://github.com/apache/spark/pull/58339#discussion_r3930476805
########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala: ########## @@ -578,12 +582,41 @@ case class CoalescedNullAwareHashPartitioning( * partitioning this one was derived from onto the same key, so one key here can * stand for several of the original ones. Sticky. See "Key Collapse" above for * what it gates and how it travels. + * @param mayContainUnknownPartitionKeys Whether the data may contain rows whose partition key is + * not among the declared `partitionKeys`. `KeyGroupedPartitioner` + * routes such rows by a deterministic hash when a side is + * re-shuffled onto this partitioning (see + * `KeyedShuffleSpec.createPartitioning`), so co-location holds + * for whole keys only: two marked partitionings declaring the + * same keys in the same order and using the same partition + * function per position still pair (equal undeclared keys hash + * to the same partition), but a row of an undeclared key sits in + * the partition of some other declared key, away from rows + * sharing a subset of its columns. `satisfies` and + * `KeyedShuffleSpec.areKeysCompatible` therefore accept a marked + * partitioning only for full-key clustering, never for a subset + * of its partition columns and never for a global ordering + * across several partitions. Two carry rules: (1) a node that + * changes the declared key set must drop the keyed partitioning, + * whether it coarsens it (key-dropping projection, reducer, Review Comment: @/tmp/r8-replies/reducer.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
