stuhood opened a new issue, #25437: URL: https://github.com/apache/datafusion/issues/25437
### Is your feature request related to a problem or challenge? When an operator tree or table scan declares `Partitioning::Range(range)`: https://github.com/apache/datafusion/blob/c4f72bad34249798182ac7de605eb63ab134f26e/datafusion/physical-expr/src/partitioning.rs#L205-L210 The partitions correspond to disjoint contiguous intervals determined by the lexicographical ordering and split points: * Partition 0: `(-inf, split_point[0])` * Partition 1: `[split_point[0], split_point[1])` * ... * Partition N: `[split_point[N-1], +inf)` When query filters or pushed-down predicates constrain the partition key (for example, `WHERE range_key < 10`, `WHERE range_key BETWEEN 20 AND 30`, or `WHERE range_key = 5`), only a subset of the partition intervals can possibly contain matching rows. However, DataFusion currently lacks plan-time static partition pruning for `RangePartitioning`. For example, in `datafusion/sqllogictest/test_files/range_partitioning.slt`: ```text 03)----FilterExec: range_key@0 < 10, projection=[non_range_key@1, value@2] 04)------DataSourceExec: file_groups=..., output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4), predicate=range_key@0 < 10, ... ``` Even though partitions 1, 2, and 3 (`[10, 20)`, `[20, 30)`, and `>= 30`) are statically known to be disjoint from `range_key < 10`: 1. `DataSourceExec` retains all 4 file groups and plans 4 partition streams. 2. `FilterExec` spawns 4 partition streams, executing scans and predicate evaluations over empty partitions. 3. Downstream operators (such as `RepartitionExec`, joins, or aggregations) schedule tasks and thread resources for all 4 partitions. 4. In distributed environments (e.g., Ballista, datafusion-distributed), scheduling tasks for statically empty partitions incurs unnecessary cluster RPCs, task dispatch overhead, and network round trips. ### Describe the solution you'd like Introduce plan-time static partition pruning for plans with `RangePartitioning`: 1. Interval evaluation: * Evaluate query filters / pushdown predicates against each partition interval `[lower_bound, upper_bound)` defined by `RangePartitioning.split_points()`. * Identify active / surviving partition indices that can evaluate to true or null. 2. Plan-time pruning: * In `ListingTable::scan` and table providers: prune unreferenced file groups so only surviving partition streams are planned. * In physical optimization (e.g., in `FilterPushdown` or a dedicated physical optimizer rule): * For pipelines requiring partition index stability (e.g. co-partitioned joins or linear merge), replace pruned partition streams with `EmptyExec` to avoid executing tasks over empty partitions while preserving partition alignment. * When downstream operators do not require matching partition alignment, prune the partition count and compact the `RangePartitioning` split points to only the active range. ### Describe alternatives you've considered * Relying solely on runtime Parquet row group / page pruning: While Parquet pruning skips row groups at runtime, it still executes physical scan tasks, schedules streams, and passes empty batches through downstream operators (filters, repartitioning, joins). * Relying on dynamic filter pushdown: Dynamic filtering operates at runtime via hash join broadcast bitsets or min/max bounds, but cannot eliminate plan-time task scheduling or optimize queries with static WHERE clauses. ### Additional context * Part of epic #25421. * Relates to #10316: while #10316 addresses runtime sequential merging and early termination for ordered streams (`SortPreservingMergeExec`), static partition pruning operates at plan time to eliminate non-overlapping partition slots for arbitrary filtered queries before execution begins. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
