stuhood opened a new issue, #27600:
URL: https://github.com/apache/datafusion/issues/27600

   ### Is your feature request related to a problem or challenge?
   
   `ordering` and `split_points` are the minimal amount of information required 
to map a row to a partition:
   
   
https://github.com/apache/datafusion/blob/51960dbad4586e41fc8995c816e6158cc77351e8/datafusion/physical-expr/src/partitioning.rs#L200-L211
   
   Recent and proposed additions attach dataset and slice properties to 
`RangePartitioning` (`samples` in #24766, and `start_bound` / `end_bound` in 
#26509 / #26593). Doing so splits partitioning equality (`PartialEq` vs 
`has_same_layout`, requiring follow-ups like #27106 while leaving bounds 
unenforced by `EnforceDistribution`), turns routing into a partial function 
with soundness hazards across joins and custom plans, cannot represent 
non-contiguous empty partitions or `Hash` slices, and restricts distribution 
synopses to plans that are already `Range`-partitioned (detailed walkthrough in 
https://github.com/apache/datafusion/pull/26593#pullrequestreview-5866547472).
   
   ### Describe the solution you'd like
   
   As an alternative to #26509 (and migrating away from 
`RangePartitioning::samples` from #24766), keep `RangePartitioning` strictly as 
`{ ordering, split_points }` and express data properties on 
[`Statistics`](https://github.com/apache/datafusion/blob/51960dbad4586e41fc8995c816e6158cc77351e8/datafusion/common/src/stats.rs)
 (`partition_statistics` via #26094's `StatisticsContext`):
   
   1. _Emptiness + `split_points` instead of slice bounds (#26509)_: advertise 
the stage-wide `Range(N, split_points)` on every task and expose unowned/empty 
partitions in the plan via `StatisticsContext` (`stats[i].num_rows == 
Precision::Exact(0)`), giving optimizer rules the task's exact responsibility 
intervals `{ [cut_i, cut_{i+1}) | stats[i].num_rows != Exact(0) }` without a 
new `Partitioning` API while letting fan-in/exchange boundaries 
(`RepartitionExec`, `CoalescePartitionsExec`, stage shuffles) skip `Exact(0)` 
inputs.
   2. _Per-partition extrema_: use per-partition `[min, max]` `Statistics` 
(extended for composite/expression keys as in #23089) to tighten observed 
bounds within those responsibility intervals for #25483 channel pruning, 
dynamic filters, and join pruning without affecting `Partitioning` equality.
   3. _Distribution synopses instead of `samples` (#27374)_: store encapsulated 
key-distribution synopses on `Statistics`, drive range scaling in 
`EnforceDistribution` and `RepartitionExec::repartitioned` from `Statistics`, 
and deprecate/remove `RangePartitioning::samples`.
   
   ### Describe alternatives you've considered
   
   Encoding slice bounds (`start_bound` / `end_bound` in #26509 / #26593) and 
distribution synopses (`samples` in #24766) directly on `RangePartitioning` 
instead of on `Statistics`:
   
   * On a `100`-worker x `16`-core (`1,600`-partition) stage, #26509 trims each 
worker's `RangePartitioning` to its `16` owned partitions (`0` empty slots) at 
the cost of the equality and soundness hazards above, whereas keeping 
stage-wide `Range(1600)` leaves `1,584` `Exact(0)` partitions per worker.
   * Comparing those two local plans head-to-head across a 7-operator join + 
exchange pipeline (with `16` non-contiguous active partitions and `1,584` 
`Exact(0)` partitions) shows that with batch 
`StatisticsContext::compute_partitions` (`0.10 ms`) and `Exact(0)` input 
elision, `Range(1600)` executes zero empty leaf streams and matches a 
`16`-partition plan within measurement noise (`0.56 ms` vs `0.70 ms` coalesce, 
`1.36 ms` vs `1.47 ms` `Hash(16)` repartition, and `833 -> 10.6 ms` `1,600`-way 
shuffle): 
https://github.com/paradedb/datafusion/blob/732c2171387105f567d66a27ccc573ddc2cbe2c8/BENCHMARK_EMPTY_PARTITIONS.md
 (commit 
https://github.com/paradedb/datafusion/commit/732c2171387105f567d66a27ccc573ddc2cbe2c8).
   
   ### Additional context
   
   * Motivation and operator/AQE walkthrough: 
https://github.com/apache/datafusion/pull/26593#pullrequestreview-5866547472
   * Related issues and PRs: #23089, #24712, #24766, #25483, #26094, #26509, 
#26593, #27106, #27374, #27384 (orthogonal `<=` split-point relaxation)
   * Downstream context: datafusion-contrib/datafusion-distributed#628, 
datafusion-contrib/datafusion-distributed#782, apache/datafusion-ballista#2364, 
apache/datafusion-ballista#2652, apache/datafusion-ballista#2722
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to