viirya commented on a change in pull request #25204: [SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery URL: https://github.com/apache/spark/pull/25204#discussion_r306391020
########## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala ########## @@ -465,19 +519,33 @@ object RewriteCorrelatedScalarSubquery extends Rule[LogicalPlan] { val alwaysTrueRef = AttributeReference(ALWAYS_TRUE_COLNAME, BooleanType)(exprId = alwaysTrueExprId) - val aggValRef = query.output.head - if (havingNode.isEmpty) { // CASE 2: Subquery with no HAVING clause + + // We replace original expression id with a new one. The added Alias column + // must use expr id of original output. If we don't replace old expr id in the + // query, the added Project in potential Project-Filter-Project can be removed + // by removeProjectBeforeFilter in ColumnPruning. Review comment: Worth trying, right now not sure if any other thing will be affected. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org