dongjoon-hyun opened a new pull request, #60210:
URL: https://github.com/apache/spark/pull/60210
### What changes were proposed in this pull request?
This PR fixes a correctness issue. In ANSI mode, decimal `div` returns a
silently wrapped `BIGINT` when the integral quotient fits its intermediate
`DECIMAL(p, 0)` type but not `BIGINT`. After this PR, it fails with
`ARITHMETIC_OVERFLOW`, which `div` on `BIGINT` already raises for
`Long.MinValue div -1`.
```sql
-- spark.sql.ansi.enabled=true (default)
-- Issue 1: It should fail with ARITHMETIC_OVERFLOW, but returns
687399551400672280.
SELECT CAST(repeat('9', 35) AS DECIMAL(35, 0)) div 0.001BD;
-- Issue 2: It should fail with ARITHMETIC_OVERFLOW, but returns
-9223372036854775808.
SELECT 9223372036854775808BD div 1;
```
`IntegralDivide` computes the decimal quotient, checks it against
`DECIMAL(p1 - s1 + s2, 0)`, and then converts it with `Decimal.toLong`. That
conversion uses `BigDecimal.longValue` and keeps only the low 64 bits. When the
decimal result type has more than 18 digits, a quotient can fit the decimal
type but not a long.
- The interpreted path (`integral.toLong(res)`) and codegen
(`$decimalValue.toLong()`) now check that the quotient fits a long. They raise
`QueryExecutionErrors.overflowInIntegralDivideError` with the query context
when it does not.
- The check runs only when `failOnError` is true (ANSI and TRY) and the
decimal result type has more than 18 digits. Smaller types always fit a long,
so their generated code does not change.
- `DivModLike.decimalToDataTypeCodeGen` now returns the assignment statement
and receives the error context code, so that `IntegralDivide` can add the
check. `Divide` and `Remainder` keep the same generated code.
- LEGACY mode keeps returning the wrapped value. It is the same as
`CAST(decimal AS BIGINT)` and `BIGINT` `div` in LEGACY mode.
### Why are the changes needed?
This is a correctness issue. ANSI mode documents that Spark throws an
arithmetic exception at runtime for numeric overflow, but decimal `div`
returned a wrong value without any error. ANSI mode is the default since Spark
4.0.0, so the default configuration returns wrong results.
The wrap was known when decimal support was added to `div` in SPARK-28322
([v3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0), 2020-06-06).
Its test pins it with the comment "overflows long and so returns a wrong
result", and the review noted that the long overflow "is yet to be handled in a
different PR"
(https://github.com/apache/spark/pull/25136#discussion_r312702907). SPARK-35152
([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0), 2021-10-06)
later added the ANSI overflow check to `div`, but only for `Long.MinValue div
-1`.
MySQL's `DIV` also converts a decimal quotient to `BIGINT` and raises an
error when the result exceeds the `BIGINT` range.
### Does this PR introduce _any_ user-facing change?
Yes. In ANSI mode (the default), decimal `div` whose quotient is outside the
`BIGINT` range now fails with `ARITHMETIC_OVERFLOW` instead of returning a
wrong value.
```
[ARITHMETIC_OVERFLOW] Overflow in integral divide. Use 'try_divide' to
tolerate overflow and return NULL instead. If necessary set
"spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22003
```
LEGACY mode is unchanged. All released versions since Spark 3.0.0 are
affected, including
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
### How was this patch tested?
Pass the CIs with a new test case in `ArithmeticExpressionSuite`. It checks
the `DECIMAL(19, 0)` boundaries `Long.MaxValue` and `Long.MinValue` and one
past each. `checkEvaluation` and `checkExceptionInExpression` cover the
interpreted and codegen paths.
Two existing tests pinned the wrapped value in the default ANSI mode, and
this PR updates them:
- `SPARK-28322: IntegralDivide supports decimal type` now runs the wrapped
case in LEGACY mode.
- The random oracle of `SPARK-45786: Decimal multiply, divide, remainder,
quot` used `quotExpected.toLong`. It now expects `ARITHMETIC_OVERFLOW` in ANSI
mode when the exact quotient is outside the long range.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
