[
https://issues.apache.org/jira/browse/SPARK-60267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Jarvin updated SPARK-60267:
--------------------------------
Affects Version/s: 4.2.0
> Segment-tree window frame fails with negative maxCachedBlocks for ROWS
> offsets near Int.MaxValue
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60267
> URL: https://issues.apache.org/jira/browse/SPARK-60267
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 5.0.0
> Reporter: Mark Jarvin
> Priority: Major
>
> h2. Summary
> With {{spark.sql.windowExec.segmentTree.enabled=true}} (default {{false}},
> added in 4.2.0), a ROWS window frame with an offset near {{Int.MaxValue}}
> fails the query with:
> {noformat}
> java.lang.IllegalArgumentException: requirement failed: maxCachedBlocks must
> be >= 1 when specified, got -32766
> at
> org.apache.spark.sql.execution.window.WindowSegmentTree.<init>(WindowSegmentTree.scala:69)
> at
> org.apache.spark.sql.execution.window.SegmentTreeWindowFunctionFrame.prepare(SegmentTreeWindowFunctionFrame.scala:165)
> {noformat}
> No large data is needed; five rows reproduce it.
> h2. Reproduction
> {code:sql}
> SET spark.sql.windowExec.segmentTree.enabled=true;
> SET spark.sql.windowExec.segmentTree.minPartitionRows=1;
> SELECT id, sum(id) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND 2147483647
> FOLLOWING)
> FROM VALUES (1), (2), (3), (4), (5) AS t(id);
> {code}
> Frames that fail with the segment tree enabled:
> * {{ROWS BETWEEN CURRENT ROW AND 2147483647 FOLLOWING}} (got -32766)
> * {{ROWS BETWEEN 2147483647 PRECEDING AND CURRENT ROW}} (got -32766)
> * {{ROWS BETWEEN 1 PRECEDING AND 2147483647 FOLLOWING}} (got -32765)
> * {{ROWS BETWEEN 2147483647 PRECEDING AND 1 FOLLOWING}} (got -32765)
> Frames that work: {{2147483647 PRECEDING AND 2147483647 FOLLOWING}}, and any
> frame with an UNBOUNDED bound. With the segment tree disabled (the default)
> all of these return correct results.
> h2. Root cause
> {{WindowEvaluatorFactoryBase.estimateMaxCachedBlocks}} computes the frame
> width in 32-bit {{Int}} arithmetic:
> {code:scala}
> case (CurrentRow, IntegerLiteral(hi)) => Some(math.abs(hi) + 1)
> case (IntegerLiteral(lo), CurrentRow) => Some(math.abs(lo) + 1)
> case (IntegerLiteral(lo), IntegerLiteral(hi)) => Some(math.abs(hi - lo) + 1)
> {code}
> For {{hi = Int.MaxValue}} the {{+ 1}} wraps to {{Int.MinValue}}, so {{ceil(w
> / blockSize) + 2}} is negative and the {{require}} in {{WindowSegmentTree}}
> fails. {{hi - lo}} can wrap the same way: for {{-2147483647 .. 2147483647}}
> it yields -2, so the budget is silently underestimated (a performance problem
> only).
> h2. Proposed fix
> Compute the width in {{Long}}, clamp the resulting block count to the {{Int}}
> range, and add a regression test that runs these frames with the segment tree
> enabled and compares against the non-segment-tree result. No existing suite
> covers huge ROWS offsets with the segment tree on.
> The failure is loud (an exception), not a silent wrong result, and the
> feature is opt-in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
