Asif created SPARK-41141:
----------------------------
Summary: avoid introducing a new aggregate expression in the
analysis phase when subquery is referencing it
Key: SPARK-41141
URL: https://issues.apache.org/jira/browse/SPARK-41141
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.3.1
Reporter: Asif
Currently the analyzer phase rules on subquery referencing the aggregate
expression in outer query, avoids introducing a new aggregate only for a single
level aggregate function. It introduces new aggregate expression for nested
aggregate functions.
It is possible to avoid adding this extra aggregate expression easily,
atleast if the outer projection involving aggregate function is exactly same as
the one that is used in subquery, or if the outer query's projection involving
aggregate function is a subtree of the subquery's expression.
Thus consider the following 2 cases:
1) select cos (sum(a)) , b from t1 group by b having exists (select x from t2
where y = cos(sum(a)) )
2) select sum(a) , b from t1 group by b having exists (select x from t2 where
y = cos(sum(a)) )
In both the above cases, there is no need for adding extra aggregate expression.
I am also investigating if its possible to avoid if the case is
3) select Cos(sum(a)) , b from t1 group by b having exists (select x from t2
where y = sum(a) )
This Jira also is needed for another issue where subquery datasource v2 is
projecting columns which are not needed. ( no Jira filed yet for that, will do
that..)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
