Mark Jarvin created SPARK-60268:
-----------------------------------
Summary: Fail window partitions larger than Int.MaxValue rows
while buffering instead of after
Key: SPARK-60268
URL: https://issues.apache.org/jira/browse/SPARK-60268
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Mark Jarvin
h2. Summary
Since SPARK-59929, window execution rejects a partition with more than
{{Int.MaxValue}} rows when any of its frames cannot handle it, using error
{{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}}. The check
({{WindowEvaluatorFactoryBase.checkPartitionSizeLimit}}) runs only after the
whole partition has been read and buffered/spilled, in
{{WindowEvaluatorFactory.fetchNextPartition}} and
{{ArrowWindowPythonEvaluatorFactory.fetchNextPartition}}.
So a query that is going to fail still reads and spills the entire oversized
partition first. The reproduction in SPARK-59929 took about 30 minutes over
2.2B rows on one core, plus the disk used by the spill, before the error
appeared.
h2. Proposal
When any frame does not support large partitions, check the buffered row count
while rows are being added (one {{Long}} comparison per row) and throw as soon
as it exceeds {{Int.MaxValue}}. Frames that do support large partitions
(LEAD/LAG and NTH_VALUE over the whole partition, see SPARK-59941) must keep
buffering.
Needs a test with a stubbed or small buffer that reports a large length, since
a real oversized partition is impractical in a unit test.
Depends on SPARK-59941 (introduces
{{WindowFunctionFrame.supportsLargePartition}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
