[ https://issues.apache.org/jira/browse/SPARK-39216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allison Wang updated SPARK-39216: --------------------------------- Description: SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't collapse projects that contain correlated subqueries since haven't been de-correlated (PullupCorrelatedPredicates). Here is a simple example to reproduce this issue {code:java} SELECT (SELECT IF(x, 1, 0)) AS a FROM (SELECT true) t(x) UNION SELECT 1 AS a {code} Exception: {code:java} java.lang.IllegalStateException: Couldn't find x#4 in [] {code} was: SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't collapse projects that contain correlated subqueries since haven't been de-correlated (PullupCorrelatedPredicates). Here is a simple example to reproduce this issue {code:java} SELECT (SELECT IF(x, 1, 0)) AS a FROM (SELECT true) t(x) UNION SELECT 1 AS a {code} Exception: {code:java} java.lang.IllegalStateException: Couldn't find x#4 in [] {code} > Issue with correlated subquery and Union > ---------------------------------------- > > Key: SPARK-39216 > URL: https://issues.apache.org/jira/browse/SPARK-39216 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.3.0 > Reporter: Allison Wang > Priority: Major > > > SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't > collapse projects that contain correlated subqueries since haven't been > de-correlated (PullupCorrelatedPredicates). > Here is a simple example to reproduce this issue > {code:java} > SELECT (SELECT IF(x, 1, 0)) AS a > FROM (SELECT true) t(x) > UNION > SELECT 1 AS a {code} > Exception: > {code:java} > java.lang.IllegalStateException: Couldn't find x#4 in [] {code} > -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org