princejain756 opened a new pull request, #26551: URL: https://github.com/apache/datafusion/pull/26551
## Which issue does this PR close? - Closes #26504. ## Rationale for this change When a correlated subquery contains an aliased derived table (`SubqueryAlias`) with a correlated predicate inside it and a `Filter` directly above the `SubqueryAlias` whose predicates are all pulled up as join filters (for example `WHERE x.d > 0`), `PullUpCorrelatedExpr::f_up` eliminates the `Filter` and rebuilds `new_plan` equal to `*plan_filter.input` (`SubqueryAlias`). Previously, `self.correlated_subquery_cols_map.insert(new_plan.clone(), correlated_subquery_cols)` overwrote the entry that `SubqueryAlias` had already recorded for `new_plan`. Because `collect_local_correlated_cols` treats `SubqueryAlias` as a leaf node and does not recurse into its input, any `Projection` or `Aggregate` above the eliminated `Filter` lost the correlated columns from the derived table and failed with `Schema error: No field named __correlated_sq_1.c`. ## What changes are included in this PR? - Use `.entry(new_plan.clone()).or_default().extend(correlated_subquery_cols)` in `PullUpCorrelatedExpr::f_up` when handling `LogicalPlan::Filter` so existing correlated columns recorded on `new_plan` are preserved when the `Filter` is removed. - Add optimizer unit tests in `decorrelate_predicate_subquery` and `scalar_subquery_to_join` and sqllogictest coverage in `subquery.slt`. ## Are these changes tested? Yes, tested via `cargo test -p datafusion-optimizer --lib` and `subquery.slt`. ## Are there any user-facing changes? Correlated `EXISTS`, `IN`, and scalar subqueries that filter an aliased derived table on an outer column now plan and execute instead of failing with a schema error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
