Mingliang Zhu created SPARK-60988:
-------------------------------------
Summary: Optimize DISTINCT window aggregate execution
Key: SPARK-60988
URL: https://issues.apache.org/jira/browse/SPARK-60988
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Mingliang Zhu
Follow-up to #SPARK-58614 ([PR
#57815|https://github.com/apache/spark/pull/57815]), covering three performance
improvements:
# *Eliminate unnecessary DISTINCT processing.* Extend {{EliminateDistinct}} to
window expressions for duplicate-insensitive aggregates such as {{{}MIN{}}},
{{{}MAX{}}}, {{{}BIT_AND{}}}, {{{}BIT_OR{}}}, and {{{}COLLECT_SET{}}}. Assess
{{FIRST}} and {{LAST}} separately because of their ordering semantics.
# *Reduce full-partition position metadata.* Avoid storing
{{{}firstVisibleIndex{}}}, which is always zero for full-partition frames.
Retain {{inputIndex}} where needed to restore first-occurrence order after
sort-based deduplication.
# *Avoid redundant event sorting on the growing-frame hash path.* Currently,
every window partition creates an event sorter even when in-memory hash entries
are already in event order. This introduces substantial initialization and
allocation overhead for many small partitions. Explore direct ordered replay,
while preserving correctness under memory pressure and ensuring proper resource
cleanup. Spill-capable destructive map iteration must not be assumed to
preserve insertion order.
Add regression tests and benchmarks covering many small partitions, large
partitions, hash/sort fallback, spills, and collations. Preserve existing
aggregate semantics, memory-control guarantees, and supported window-frame
restrictions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
