Shahzeb-DataEngineer opened a new issue, #26648:
URL: https://github.com/apache/datafusion/issues/26648
### Describe the bug
`sum()` over a `DECIMAL` column silently wraps around on overflow and
returns a wrong result, instead of raising an arithmetic overflow error.
Summing two positive `DECIMAL(38, 0)` values near the type maximum returns a
*negative* number. The returned value also has 39 digits, which violates the
declared result type of the aggregate (`Decimal128(38, 0)`).
This is inconsistent with the rest of the engine for the same overflow:
* scalar decimal `+` and `*` raise `Arrow error: Arithmetic overflow`
* `avg()` over the same `Decimal256` data raises `Arithmetic Overflow in
AvgAccumulator`
Only `sum()` wraps, and it does so silently.
### To Reproduce
```sql
SELECT arrow_typeof(sum(v)), sum(v)
FROM (VALUES (CAST('99999999999999999999999999999999999999' AS
DECIMAL(38,0))),
(CAST('99999999999999999999999999999999999999' AS
DECIMAL(38,0)))) t(v);
-- Decimal128(38, 0) | -140282366920938463463374607431768211458 <--
wrapped, 39 digits
SELECT k, sum(v) FROM (VALUES (1,
CAST('99999999999999999999999999999999999999' AS DECIMAL(38,0))),
(1,
CAST('99999999999999999999999999999999999999' AS DECIMAL(38,0))),
(2, CAST('5' AS DECIMAL(38,0)))) t(k,v) GROUP
BY k ORDER BY k;
-- 1 | -140282366920938463463374607431768211458
-- 2 | 5
```
The same wrapping reproduces for `DECIMAL(76, 0)` / `Decimal256` (seven rows
of 76 nines):
```sql
SELECT sum(v) FROM ...;
--
-45792089237316195423570985008687907853269984665640564039457584007913129639943
(wrapped)
SELECT avg(v) FROM ...;
-- Execution error: Arithmetic Overflow in AvgAccumulator
```
### Expected behavior
`sum()` over a decimal column should raise an arithmetic overflow error when
the true sum is not representable in the result type, consistent with scalar
decimal arithmetic and with `avg()`.
Note that this report is specifically about *decimal* sums. Integer `sum()`
wrapping matches DataFusion's documented lenient non-ANSI integer arithmetic
behavior, and I am not proposing to change integer behavior.
### Additional context
The decimal paths of the SUM accumulator in
`datafusion/functions-aggregate/src/sum.rs` use wrapping addition
(`add_wrapping`) in the groups accumulator, `update_batch`, the sliding
accumulator, and `retract_batch`. Batch partials also come from
`arrow::compute::sum`, which wraps as well.
Willingness to contribute: I plan to contribute a fix for this bug
independently, and will open a PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
