Vladimir Golubev created SPARK-58503:
----------------------------------------
Summary: In the presence of a generator function, Having is not
pushed below the Window
Key: SPARK-58503
URL: https://issues.apache.org/jira/browse/SPARK-58503
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Vladimir Golubev
SQL (with a no-op generator):
SELECT explode(array(a)) AS col, count(*) OVER () AS cnt, a
FROM VALUES (1), (2), (3), (NULL) AS t(a)
GROUP BY a
HAVING a IS NOT NULL;
->
col cnt a
1 4 1
2 4 2
3 4 3
Plan:
col: int, cnt: bigint, a: int
Filter isnotnull(a#13438)
+- Project [col#13450, cnt#13440L, a#13438]
+- Generate explode(_gen_input_0#13448), false, [col#13450]
+- Window [array(a#13438) AS _gen_input_0#13448, count(1)
windowspecdefinition(specifiedwindowframe(RowFrame, unboundedpreceding$(),
unboundedfollowing$())) AS cnt#13440L, a#13438]
+- Aggregate [a#13438], [a#13438]
+- SubqueryAlias t
+- LocalRelation [a#13438]
Equivalently, if there was no no-op generator explode(array(a)):
SELECT a AS col, count(*) OVER () AS cnt, a
FROM VALUES (1), (2), (3), (NULL) AS t(a)
GROUP BY a
HAVING a IS NOT NULL;
->
col cnt a
1 3 1
2 3 2
3 3 3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
