[
https://issues.apache.org/jira/browse/SPARK-38162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-38162:
------------------------------
Description:
Optimize the plan if its max row is equal to or less than 1 in these cases:
* if sort max rows less than or equal to 1, remove the sort
* if local sort max rows per partition less than or equal to 1, remove the
local sort
* if aggregate max rows less than or equal to 1 and it's grouping only, remove
the aggregate
* if aggregate max rows less than or equal to 1, set distinct to false in all
aggregate expression
was:We can not propagate empty through aggregate if it does not contain
grouping expression. But for the aggregate which contains distinct aggregate
expression, we can remove distinct if its child is empty.
> Optimize one max row plan in normal and AQE Optimizer
> -----------------------------------------------------
>
> Key: SPARK-38162
> URL: https://issues.apache.org/jira/browse/SPARK-38162
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: XiDuo You
> Priority: Major
>
> Optimize the plan if its max row is equal to or less than 1 in these cases:
> * if sort max rows less than or equal to 1, remove the sort
> * if local sort max rows per partition less than or equal to 1, remove the
> local sort
> * if aggregate max rows less than or equal to 1 and it's grouping only,
> remove the aggregate
> * if aggregate max rows less than or equal to 1, set distinct to false in
> all aggregate expression
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
