stuhood opened a new issue, #27600: URL: https://github.com/apache/datafusion/issues/27600
### Is your feature request related to a problem or challenge? `ordering` and `split_points` are the minimal amount of information required to map a row to a partition: https://github.com/apache/datafusion/blob/51960dbad4586e41fc8995c816e6158cc77351e8/datafusion/physical-expr/src/partitioning.rs#L200-L211 Recent and proposed additions attach dataset and slice properties to `RangePartitioning` (`samples` in #24766, and `start_bound` / `end_bound` in #26509 / #26593). Doing so splits partitioning equality (`PartialEq` vs `has_same_layout`, requiring follow-ups like #27106 while leaving bounds unenforced by `EnforceDistribution`), turns routing into a partial function with soundness hazards across joins and custom plans, cannot represent non-contiguous empty partitions or `Hash` slices, and restricts distribution synopses to plans that are already `Range`-partitioned (detailed walkthrough in https://github.com/apache/datafusion/pull/26593#pullrequestreview-5866547472). ### Describe the solution you'd like As an alternative to #26509 (and migrating away from `RangePartitioning::samples` from #24766), keep `RangePartitioning` strictly as `{ ordering, split_points }` and express data properties on [`Statistics`](https://github.com/apache/datafusion/blob/51960dbad4586e41fc8995c816e6158cc77351e8/datafusion/common/src/stats.rs) (`partition_statistics` via #26094's `StatisticsContext`): 1. _Emptiness + `split_points` instead of slice bounds (#26509)_: advertise the stage-wide `Range(N, split_points)` on every task and expose unowned/empty partitions in the plan via `StatisticsContext` (`stats[i].num_rows == Precision::Exact(0)`), giving optimizer rules the task's exact responsibility intervals `{ [cut_i, cut_{i+1}) | stats[i].num_rows != Exact(0) }` without a new `Partitioning` API while letting fan-in/exchange boundaries (`RepartitionExec`, `CoalescePartitionsExec`, stage shuffles) skip `Exact(0)` inputs. 2. _Per-partition extrema_: use per-partition `[min, max]` `Statistics` (extended for composite/expression keys as in #23089) to tighten observed bounds within those responsibility intervals for #25483 channel pruning, dynamic filters, and join pruning without affecting `Partitioning` equality. 3. _Distribution synopses instead of `samples` (#27374)_: store encapsulated key-distribution synopses on `Statistics`, drive range scaling in `EnforceDistribution` and `RepartitionExec::repartitioned` from `Statistics`, and deprecate/remove `RangePartitioning::samples`. ### Describe alternatives you've considered Encoding slice bounds (`start_bound` / `end_bound` in #26509 / #26593) and distribution synopses (`samples` in #24766) directly on `RangePartitioning` instead of on `Statistics`: * On a `100`-worker x `16`-core (`1,600`-partition) stage, #26509 trims each worker's `RangePartitioning` to its `16` owned partitions (`0` empty slots) at the cost of the equality and soundness hazards above, whereas keeping stage-wide `Range(1600)` leaves `1,584` `Exact(0)` partitions per worker. * Comparing those two local plans head-to-head across a 7-operator join + exchange pipeline (with `16` non-contiguous active partitions and `1,584` `Exact(0)` partitions) shows that with batch `StatisticsContext::compute_partitions` (`0.10 ms`) and `Exact(0)` input elision, `Range(1600)` executes zero empty leaf streams and matches a `16`-partition plan within measurement noise (`0.56 ms` vs `0.70 ms` coalesce, `1.36 ms` vs `1.47 ms` `Hash(16)` repartition, and `833 -> 10.6 ms` `1,600`-way shuffle): https://github.com/paradedb/datafusion/blob/732c2171387105f567d66a27ccc573ddc2cbe2c8/BENCHMARK_EMPTY_PARTITIONS.md (commit https://github.com/paradedb/datafusion/commit/732c2171387105f567d66a27ccc573ddc2cbe2c8). ### Additional context * Motivation and operator/AQE walkthrough: https://github.com/apache/datafusion/pull/26593#pullrequestreview-5866547472 * Related issues and PRs: #23089, #24712, #24766, #25483, #26094, #26509, #26593, #27106, #27374, #27384 (orthogonal `<=` split-point relaxation) * Downstream context: datafusion-contrib/datafusion-distributed#628, datafusion-contrib/datafusion-distributed#782, apache/datafusion-ballista#2364, apache/datafusion-ballista#2652, apache/datafusion-ballista#2722 ### Willingness to contribute I plan to contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
