[
https://issues.apache.org/jira/browse/SPARK-58503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58503:
-----------------------------------
Labels: pull-request-available (was: )
> In the presence of a generator function, Having is not pushed below the Window
> ------------------------------------------------------------------------------
>
> Key: SPARK-58503
> URL: https://issues.apache.org/jira/browse/SPARK-58503
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Vladimir Golubev
> Priority: Major
> Labels: pull-request-available
>
> SQL (with a no-op generator):
> SELECT explode(array(a)) AS col, count(*) OVER () AS cnt, a
> FROM VALUES (1), (2), (3), (NULL) AS t(a)
> GROUP BY a
> HAVING a IS NOT NULL;
> ->
> col cnt a
> 1 4 1
> 2 4 2
> 3 4 3
> Plan:
> col: int, cnt: bigint, a: int
> Filter isnotnull(a#13438)
> +- Project [col#13450, cnt#13440L, a#13438]
> +- Generate explode(_gen_input_0#13448), false, [col#13450]
> +- Window [array(a#13438) AS _gen_input_0#13448, count(1)
> windowspecdefinition(specifiedwindowframe(RowFrame, unboundedpreceding$(),
> unboundedfollowing$())) AS cnt#13440L, a#13438]
> +- Aggregate [a#13438], [a#13438]
> +- SubqueryAlias t
> +- LocalRelation [a#13438]
> Equivalently, if there was no no-op generator explode(array(a)):
> SELECT a AS col, count(*) OVER () AS cnt, a
> FROM VALUES (1), (2), (3), (NULL) AS t(a)
> GROUP BY a
> HAVING a IS NOT NULL;
> ->
> col cnt a
> 1 3 1
> 2 3 2
> 3 3 3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
