mohitgurav20 commented on issue #25792: URL: https://github.com/apache/datafusion/issues/25792#issuecomment-5855359684
I took a deep dive into `PullUpCorrelatedExpr` in `datafusion/optimizer/src/decorrelate.rs` to trace why correlated filters under window functions get pulled up into decorrelated joins. ### Root Cause During `f_up` traversal in `PullUpCorrelatedExpr`: 1. `LogicalPlan::Filter` extracts the outer-referencing predicates into `self.join_filters`. 2. When bottom-up rewriting reaches `LogicalPlan::Window`, the match statement in `f_up` currently lacks an arm for `Window`. It falls through to `_ => Ok(Transformed::no(plan))` while leaving `can_pull_up = true`. 3. The correlated predicate gets pulled past the window operator into the generated `Join`. Consequently, window evaluation runs over the entire inner relation prior to join filtering, distorting rank/row count/aggregation results. ### Proposed Fix We should handle `LogicalPlan::Window` in `f_up`: * Check if join filters or IN-predicates were pulled up from below (`!self.join_filters.is_empty() || self.in_predicate_opt.is_some()`). * Check whether the window's `PARTITION BY` expressions cover all inner columns referenced in the pulled-up join filters. * If the correlated columns are **not** covered by `PARTITION BY` (or if `PARTITION BY` is empty), set `self.can_pull_up = false`. This gracefully prevents invalid decorrelation and falls back to scalar/unsupported subquery handling. I can open a PR with the fix and regression test cases in `sqllogictest` if you'd like to assign this to me! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
