mihailotim-db commented on code in PR #54729:
URL: https://github.com/apache/spark/pull/54729#discussion_r2944971176
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ExpressionIdAssigner.scala:
##########
@@ -597,11 +700,126 @@ class ExpressionIdAssigner {
}
}
+ /**
+ * Insert the subquery expression id to the current mapping.
+ * Some optimization rules rely on subquery expression ids such as
[[mergeScalarSubqueries]].
+ * And some subquery expression id may be referenced in other operators.
(e.g.: [[Exists]]
+ * subquery expression id may be referenced as an attribute reference in
[[Filter]] condition).
+ * It is important to avoid duplicated subquery expression ids and add the
(oldId -> newId) into
+ * the current mapping to make sure later references is correct if the
subquery expression id is
+ * reassigned.
+ *
+ * There are also cases where the same subquery expression can appear again
in the same operator
+ * instead of being referenced as an [[AttributeReference]] in the latter
usage. For example,
+ * consider the following query:
+ * {{{
+ * select country,
+ * (select count(*) from data) as cnt,
+ * count(id) as cnt_id from data group by all
+ * }}}
+ * The logical plan for it is:
+ * {{{
+ * Aggregate [country#30, scalar-subquery#27 []],
+ * [country#30, scalar-subquery#27 [] AS cnt#28L, count(1) AS cnt_id#29L]
+ * :- Aggregate [count(1) AS count(1)#52L]
+ * : +- Project
+ * : +- LocalRelation [country#40, city#41, name#42, id#43, power#44]
+ * +- Aggregate [count(1) AS count(1)#52L]
+ * +- Project
+ * +- LocalRelation [country#53, city#54, name#55, id#56, power#57]
+ * +- Project [country#30]
+ * +- LocalRelation [country#30, city#31, name#32, id#33, power#34]
+ * }}}
+ * In this case, we reassign all the attributes within the subquery plan
together with the
+ * subquery expression itself to avoid duplicate expression ids.
+ */
+ def mapSubqueryExpression[SubqueryExpressionType <: SubqueryExpression](
+ subquery: SubqueryExpression,
+ alwaysUpdateExpressionId: Boolean = false): SubqueryExpressionType = {
+ if (mappingStack.peek().mapping.isEmpty) {
+ throw SparkException.internalError(
+ "Expression ID mapping doesn't exist. Please first call " +
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
