avantgardnerio opened a new issue, #2220: URL: https://github.com/apache/datafusion-ballista/issues/2220
## Background Every `ExchangeExec` in the AQE plan carries two independently-resolvable slots: - **coalesce** (`CoalescePlan`): set by `CoalescePartitionsRule` based on runtime sub-part size heuristics — decides post-coalesce partition count K. - **range-repartition routing** (`RangeRepartitionRouting`): set by `SchedulerAqe` when a range-repartitioned upstream stage completes and its cuts have been merged — carries `cuts` (length = K − 1). ## Bug Both slots can land on the same `ExchangeExec`. `CoalescePartitionsRule::optimize` walks leaf exchanges and bails only on `broadcast`; it has no equivalent bail for range-repartitioned. `SchedulerAqe::repartition_routing` sets the routing slot regardless of what `CoalescePartitionsRule` has already committed. When both are set, `BallistaAdapter::adapt_to_ballista`'s `(Some(cp), false)` arm builds a reader with `cp.groups.len()` partitions, then hands `cuts.len() + 1` predicates to `PerPartitionFilterExec::try_new` — which rejects the count mismatch. Hard error at plan time rather than wrong data, which makes this the lesser of two issues, but the invariant isn't documented anywhere and nothing enforces it. Reference: https://github.com/apache/datafusion-ballista/pull/2196#discussion_r3705634917 ## Proposed fixes Two viable directions: 1. **Bail (short-term)** — mirror the broadcast bail: `CoalescePartitionsRule` skips leaf groups that have any range-repartitioned exchange. Straightforward, mutually exclusive with contiguous-group coalescing for range-partitioned data, but explicit. 2. **Merge (long-term)** — coalesce groups neighbouring upstream partitions only, and range-repartition cuts are monotonic in partition order, so contiguous-group coalescing is compatible with range partitioning: merge adjacent cut buckets into one when their upstream sub-parts get grouped. Real fix; requires teaching `CoalescePartitionsRule` (or the routing resolver) about the other slot. ## Scope Filing (1) as the immediate PR button-up so the two rules aren't silently mutually exclusive. (2) is the tracked follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
