sepuri sai krishna created SPARK-61623:
------------------------------------------
Summary: Segment-tree window SUM fails with ARITHMETIC_OVERFLOW
although no frame overflows
Key: SPARK-61623
URL: https://issues.apache.org/jira/browse/SPARK-61623
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: sepuri sai krishna
With {{spark.sql.window.segmentTree.enabled=true}}, a moving-frame {{sum}} over
a BIGINT column fails with {{ARITHMETIC_OVERFLOW}} under ANSI mode although the
sum of every frame is in range. The query succeeds when the flag is off.
{code:sql}
CREATE TEMPORARY VIEW t AS
SELECT id AS i, CAST(1000000000000000 AS BIGINT) AS v FROM range(0, 100000);
SET spark.sql.window.segmentTree.enabled=false;
SELECT max(x) FROM (
SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS x
FROM t);
-- 2000000000000000
SET spark.sql.window.segmentTree.enabled=true;
SELECT max(x) FROM (
SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS x
FROM t);
-- [ARITHMETIC_OVERFLOW] overflow. Use 'try_add' to tolerate overflow and
return NULL instead. ...
{code}
Every frame holds two rows, so its sum is 2e15, far below the BIGINT maximum of
about 9.2e18. Sums of large BIGINT values over short frames are realistic, for
example over epoch times in nanoseconds (about 1.8e18 each, so that the sum of
any six rows overflows).
*Cause*
{{WindowSegmentTree}} aggregates rows that no frame covers together:
* {{computeBlockAggregates}} computes, when the tree is built, the aggregate of
every block of {{spark.sql.window.segmentTree.blockSize}} (65536) rows. In the
example one block sums to 6.5e19.
* {{buildBlockLevels}} computes the aggregate of every internal node of a block
(16, 256, 4096, ... rows).
For an integral {{Sum}} under ANSI, the buffer is a BIGINT that is added with
an exact addition, so such an aggregate can overflow and fail the query, even
though no frame reads it. {{try_sum}}, {{sum}} over DECIMAL (overflow is
checked at the end) and over DOUBLE, and non-ANSI mode are not affected.
*Fix*
Do not fail on an arithmetic error of a block root or of an internal node: keep
the node as unavailable, and let a query that covers it merge its children
instead, down to the rows. A frame whose own sum overflows still fails, as its
values are merged into the result.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]