[
https://issues.apache.org/jira/browse/SPARK-60268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Jarvin updated SPARK-60268:
--------------------------------
Priority: Minor (was: Major)
> Fail window partitions larger than Int.MaxValue rows while buffering instead
> of after
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60268
> URL: https://issues.apache.org/jira/browse/SPARK-60268
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Mark Jarvin
> Priority: Minor
>
> h2. Summary
> Since SPARK-59929, window execution rejects a partition with more than
> {{Int.MaxValue}} rows when any of its frames cannot handle it, using error
> {{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}}. The check
> ({{WindowEvaluatorFactoryBase.checkPartitionSizeLimit}}) runs only after the
> whole partition has been read and buffered/spilled, in
> {{WindowEvaluatorFactory.fetchNextPartition}} and
> {{ArrowWindowPythonEvaluatorFactory.fetchNextPartition}}.
> So a query that is going to fail still reads and spills the entire oversized
> partition first. The reproduction in SPARK-59929 took about 30 minutes over
> 2.2B rows on one core, plus the disk used by the spill, before the error
> appeared.
> h2. Proposal
> When any frame does not support large partitions, check the buffered row
> count while rows are being added (one {{Long}} comparison per row) and throw
> as soon as it exceeds {{Int.MaxValue}}. Frames that do support large
> partitions (LEAD/LAG and NTH_VALUE over the whole partition, see SPARK-59941)
> must keep buffering.
> Needs a test with a stubbed or small buffer that reports a large length,
> since a real oversized partition is impractical in a unit test.
> Depends on SPARK-59941 (introduces
> {{WindowFunctionFrame.supportsLargePartition}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
