Zikun created SPARK-32947:
-----------------------------
Summary: Support top-N sort for Spark SQL window function
Key: SPARK-32947
URL: https://issues.apache.org/jira/browse/SPARK-32947
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.1
Reporter: Zikun
In Spark SQL, there are two types of sort execution, *_SortExec_* and
*_TakeOrderedAndProjectExec_* . *_SortExec_* is a general sorting execution and
it does not support top-N sort. *_TakeOrderedAndProjectExec_* is the execution
for top-N sort in Spark. Spark SQL rank window function needs to sort the data
locally and it relies on the execution plan *_SortExec_* to sort the data in
each physical data partition. When the filter of the window rank (e.g. rank <=
100) is specified in a user's query, the filter can actually be pushed down to
the SortExec and then we let SortExec operates top-N sort. Right now SortExec
does not support top-N sort and we need to extend the capability of SortExec to
support top-N sort.
This Jira has dependency on another existing Jira
https://issues.apache.org/jira/browse/SPARK-32096
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
