Mark Jarvin created SPARK-60267:
-----------------------------------
Summary: Segment-tree window frame fails with negative
maxCachedBlocks for ROWS offsets near Int.MaxValue
Key: SPARK-60267
URL: https://issues.apache.org/jira/browse/SPARK-60267
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Mark Jarvin
h2. Summary
With {{spark.sql.windowExec.segmentTree.enabled=true}} (default {{false}},
added in 4.2.0), a ROWS window frame with an offset near {{Int.MaxValue}} fails
the query with:
{noformat}
java.lang.IllegalArgumentException: requirement failed: maxCachedBlocks must be
>= 1 when specified, got -32766
at
org.apache.spark.sql.execution.window.WindowSegmentTree.<init>(WindowSegmentTree.scala:69)
at
org.apache.spark.sql.execution.window.SegmentTreeWindowFunctionFrame.prepare(SegmentTreeWindowFunctionFrame.scala:165)
{noformat}
No large data is needed; five rows reproduce it.
h2. Reproduction
{code:sql}
SET spark.sql.windowExec.segmentTree.enabled=true;
SET spark.sql.windowExec.segmentTree.minPartitionRows=1;
SELECT id, sum(id) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND 2147483647
FOLLOWING)
FROM VALUES (1), (2), (3), (4), (5) AS t(id);
{code}
Frames that fail with the segment tree enabled:
* {{ROWS BETWEEN CURRENT ROW AND 2147483647 FOLLOWING}} (got -32766)
* {{ROWS BETWEEN 2147483647 PRECEDING AND CURRENT ROW}} (got -32766)
* {{ROWS BETWEEN 1 PRECEDING AND 2147483647 FOLLOWING}} (got -32765)
* {{ROWS BETWEEN 2147483647 PRECEDING AND 1 FOLLOWING}} (got -32765)
Frames that work: {{2147483647 PRECEDING AND 2147483647 FOLLOWING}}, and any
frame with an UNBOUNDED bound. With the segment tree disabled (the default) all
of these return correct results.
h2. Root cause
{{WindowEvaluatorFactoryBase.estimateMaxCachedBlocks}} computes the frame width
in 32-bit {{Int}} arithmetic:
{code:scala}
case (CurrentRow, IntegerLiteral(hi)) => Some(math.abs(hi) + 1)
case (IntegerLiteral(lo), CurrentRow) => Some(math.abs(lo) + 1)
case (IntegerLiteral(lo), IntegerLiteral(hi)) => Some(math.abs(hi - lo) + 1)
{code}
For {{hi = Int.MaxValue}} the {{+ 1}} wraps to {{Int.MinValue}}, so {{ceil(w /
blockSize) + 2}} is negative and the {{require}} in {{WindowSegmentTree}}
fails. {{hi - lo}} can wrap the same way: for {{-2147483647 .. 2147483647}} it
yields -2, so the budget is silently underestimated (a performance problem
only).
h2. Proposed fix
Compute the width in {{Long}}, clamp the resulting block count to the {{Int}}
range, and add a regression test that runs these frames with the segment tree
enabled and compares against the non-segment-tree result. No existing suite
covers huge ROWS offsets with the segment tree on.
The failure is loud (an exception), not a silent wrong result, and the feature
is opt-in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
