Phoenix500526 opened a new issue, #23436: URL: https://github.com/apache/datafusion/issues/23436
### Is your feature request related to a problem or challenge? Follow-up from https://github.com/apache/datafusion/pull/23189 File scans for Hive/value-partitioned data currently represent their output partitioning as `Partitioning::Hash` on the partition columns. This lets the optimizer avoid repartitioning for key-partitioned operators, but the data was not actually assigned to partitions by DataFusion's hash partitioner. This makes `Partitioning::Hash` carry multiple meanings: - DataFusion hash partitioning with a particular hash implementation / seed - another hash implementation / seed - Hive/value-based file grouping by partition column values - potentially other source-defined key partitioning methods These partitioning methods can all provide key colocation, but they are not necessarily co-partitioned by partition index. Treating them all as `Partitioning::Hash` can make later optimizer decisions and dynamic filtering decisions brittle or incorrect if they rely on partition compatibility. ### Describe the solution you'd like Discuss and design a more precise representation for value-based physical partitioning. Possible directions from the PR discussion include: - add a new `Partitioning::Value` / `ValuePartitioned` variant - represent Hive/value partitioning as `Partitioning::Range` when values and ordering are available - make the partitioning method explicit, e.g. hash algorithm/seed vs column value grouping - explore whether `Partitioning` should become extensible for source-defined partitioning methods The design should distinguish: - satisfying a `Distribution::KeyPartitioned` requirement, where key colocation may be enough - proving co-partitioning / partition-index compatibility between two inputs, where the exact partitioning method and value-to-partition mapping matter ### Describe alternatives you've considered _No response_ ### Additional context Related issues and PRs: - #19090 - #23099 - #23236 - #23266 - #23189 Relevant review thread: - https://github.com/apache/datafusion/pull/23189#discussion_r3544046751 - https://github.com/apache/datafusion/pull/23189#discussion_r3544133348 - https://github.com/apache/datafusion/pull/23189#discussion_r3549863005 - https://github.com/apache/datafusion/pull/23189#discussion_r3549906872 - https://github.com/apache/datafusion/pull/23189#discussion_r3554849635 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
