[ https://issues.apache.org/jira/browse/SPARK-25037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571032#comment-16571032 ]
Chris O'Hara commented on SPARK-25037: -------------------------------------- You're right – we ran into the issue with a custom transformation rule. I see that catalyst uses [a dedicated rule|https://github.com/apache/spark/blob/136588e95f69923a04458abe4862d336e5244c84/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L164] to optimize subquery plans, so maybe `transformAllExpressions()` was never supposed to recurse into nested plans. > plan.transformAllExpressions() doesn't transform expressions in nested > SubqueryExpression plans > ----------------------------------------------------------------------------------------------- > > Key: SPARK-25037 > URL: https://issues.apache.org/jira/browse/SPARK-25037 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.1 > Reporter: Chris O'Hara > Priority: Minor > > Given the following LogicalPlan: > {code:java} > scala> val plan = spark.sql("SELECT 1 bar FROM (SELECT 1 foo) WHERE foo IN > (SELECT 1 foo)").queryExecution.logical > plan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan = > 'Project [1 AS bar#29] > +- 'Filter 'foo IN (list#31 []) > : +- Project [1 AS foo#30] > : +- OneRowRelation > +- SubqueryAlias __auto_generated_subquery_name > +- Project [1 AS foo#28] > +- OneRowRelation > {code} > the following transformation should replace all instances of lit(1) with > lit(2): > {code:java} > scala> plan.transformAllExpressions { case l @ Literal(1, _) => l.copy(value > = 2) } > res0: plan.type = > 'Project [2 AS bar#29] > +- 'Filter 'foo IN (list#31 []) > : +- Project [1 AS foo#30] > : +- OneRowRelation > +- SubqueryAlias __auto_generated_subquery_name > +- Project [2 AS foo#28] > +- OneRowRelation > {code} > Instead, the nested SubqueryExpression plan is not transformed. > The expected output is: > {code:java} > 'Project [2 AS bar#29] > +- 'Filter 'foo IN (list#31 []) > : +- Project [2 AS foo#30] > : +- OneRowRelation > +- SubqueryAlias __auto_generated_subquery_name > +- Project [2 AS foo#28] > +- OneRowRelation > {code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org