viirya opened a new pull request, #59335:
URL: https://github.com/apache/spark/pull/59335
### What changes were proposed in this pull request?
In the BigDecimal branch of `Decimal.changePrecision`, when the scale change
exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits
(`precision - scale`, computed in Long) is checked before calling `setScale`:
- If the value is nonzero and has more integral digits than the target type
allows, return `false` (overflow) directly. Rounding cannot bring it back into
range.
- If the value is zero or below 0.01 ulp of the target scale, the result
depends only on the sign and the rounding mode, so a same-sign value of 0.01
ulp is rescaled instead. This gives the same result for every rounding mode.
Otherwise the remaining scale change is bounded by the number of digits of
the value, so `setScale` is unchanged. The extra check costs one Int comparison
when the scale change is small, which is the common case.
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal with a
negative scale (only possible under
`spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of
`toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a
String. `QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical
copy, now delegates to `DataTypeErrors.cannotChangeDecimalPrecisionError`.
### Why are the changes needed?
`setScale` takes time proportional to the scale change and throws once it
exceeds the Int range:
```sql
SELECT CAST('1e-2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: BigInteger would overflow supported range
(LEGACY, ANSI and TRY)
SELECT CAST('1e-100000000' AS DECIMAL(10,2));
-- 0.00, after ~40s
SET spark.sql.legacy.allowNegativeScaleOfDecimal=true;
SELECT CAST('1e2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: Underflow
```
This is the small-side counterpart of the fast-fail added in
SPARK-35841/SPARK-37451. With negative scales allowed, `Decimal.fromString`
intentionally skips that fast-fail, so huge values reach `changePrecision` too.
### Does this PR introduce _any_ user-facing change?
Yes.
- Casting a string whose value is far below the target scale's ulp now
returns the rounded value (e.g. 0.00) instead of throwing
`ArithmeticException`, and returns quickly.
- With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string
with a huge exponent returns NULL (non-ANSI, try_cast) or throws
`NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException:
Underflow`. In that error, a value with a negative scale is shown in scientific
notation (e.g. `1E+40`) instead of plain notation.
No input that previously produced a value changes its result.
### How was this patch tested?
New tests in `DecimalSuite`, `CastSuiteBase` (so they run in
`CastWithAnsiOffSuite`, `CastWithAnsiOnSuite` and `TryCastSuite`),
`CastWithAnsiOffSuite` and `CastWithAnsiOnSuite`, covering interpreted and
codegen evaluation. All of them fail without the fix. `DecimalSuite` also
checks that values around the bounds of both shortcuts, with short and 81-digit
mantissas, under every supported rounding mode and with and without negative
scales, round exactly like `BigDecimal.setScale`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
