[
https://issues.apache.org/jira/browse/SPARK-60269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Jarvin updated SPARK-60269:
--------------------------------
Priority: Minor (was: Major)
> Support or document window partitions larger than Int.MaxValue rows for the
> remaining window frames
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-60269
> URL: https://issues.apache.org/jira/browse/SPARK-60269
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Mark Jarvin
> Priority: Minor
>
> h2. Summary
> After SPARK-59929 and SPARK-59941, LEAD/LAG and NTH_VALUE over the whole
> partition return correct results for partitions with more than
> {{Int.MaxValue}} rows. Every other window frame still fails fast with
> {{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}} because it tracks the row
> index or window bounds in 32-bit {{Int}}s. This ticket tracks what it would
> take to support them, and which ones should stay rejected. It is low
> priority: it only affects workloads with a single partition over 2^31 rows,
> and the current behaviour is a clear error, not wrong results.
> h2. Sub-items
> *1. Plain aggregate frames (most useful).* Whole-partition aggregates (for
> example {{sum(x) OVER (PARTITION BY k)}}) and running aggregates (UNBOUNDED
> PRECEDING to CURRENT ROW). Needs:
> * {{WindowFunctionFrame.write(index: Int, ...)}} and the drivers'
> {{rowIndex}} widened to {{Long}} ({{WindowEvaluatorFactory}}).
> * {{BoundOrdering.compare}} taking {{Long}} indexes.
> * {{AggregateProcessor.initialize(size: Int)}} taking the partition size as
> {{Long}}.
> Sliding frames additionally need the frame bounds widened, and can follow.
> *2. Size-based functions: NTILE, CUME_DIST, PERCENT_RANK.* They keep the
> partition size in an {{INT}} aggregate buffer column
> ({{SizeBasedWindowFunction.n}}), so this needs a buffer schema change.
> Separate from item 1.
> *3. Frames and functions that should stay rejected (document, improve the
> message).*
> * {{ROW_NUMBER}}, {{RANK}}, {{DENSE_RANK}} return {{INT}}, so a result above
> {{Int.MaxValue}} cannot be represented.
> * UNBOUNDED FOLLOWING frames recompute the aggregate for every row
> (quadratic) and use {{getIterator(int startIndex)}}; not usable at this size.
> * The Python (Arrow) window path sends the window bounds to Python as 32-bit
> ints.
> * {{WindowSegmentTree}} indexes rows with {{Int}}s.
> The deliverable here is a clearer error message and a documentation note, not
> code.
> Related: SPARK-59929, SPARK-59941.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
