viirya opened a new pull request, #59513:
URL: https://github.com/apache/spark/pull/59513
### What changes were proposed in this pull request?
Backport of #59335 to branch-4.2.
The cherry-pick conflicted only on `CastUtils.java`, which does not exist on
this branch: `Cast` calls
`QueryExecutionErrors.cannotChangeDecimalPrecisionError` directly in both the
interpreted and generated code. So instead of removing that method and
switching `CastUtils`, this backport keeps it and makes it delegate to
`DataTypeErrors.cannotChangeDecimalPrecisionError`, which gives the same error
message. The rest of the change is identical to #59335.
In the BigDecimal branch of `Decimal.changePrecision`, when the scale change
exceeds `DecimalType.MAX_PRECISION`, the value's number of integral digits
(`precision - scale`, computed in Long) is checked before calling `setScale`:
- If the value is nonzero and has more integral digits than the target type
allows, return `false` (overflow) directly. Rounding cannot bring it back into
range.
- If the value is below 0.01 ulp of the target scale, the result depends
only on the sign and the rounding mode, so a same-sign value of 0.01 ulp is
rescaled instead. This gives the same result for every rounding mode.
The Long branch of `changePrecision` now computes the scale change in Long,
so it no longer wraps and throws `ArrayIndexOutOfBoundsException` from
`POW_10(diff)` when negative scales are allowed.
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` now renders a decimal whose
scale is below -1000 (only possible under
`spark.sql.legacy.allowNegativeScaleOfDecimal`) with `toString` instead of
`toPlainString`, since the plain form of e.g. `1E+2147483647` does not fit in a
String. Other values are rendered as before.
`QueryExecutionErrors.cannotChangeDecimalPrecisionError`, an identical copy,
now delegates to `DataTypeErrors.cannotChangeDecimalPrecisionError`.
### Why are the changes needed?
`setScale` takes time that grows with the scale change, and throws once
10^change exceeds BigInteger's range or the change exceeds the Int range:
```sql
SELECT CAST('1e-2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: BigInteger would overflow supported range
(LEGACY, ANSI and TRY)
SELECT CAST('1e-100000000' AS DECIMAL(10,2));
-- 0.00, after ~40s
SET spark.sql.legacy.allowNegativeScaleOfDecimal=true;
SELECT CAST('1e2147483647' AS DECIMAL(10,2));
-- java.lang.ArithmeticException: Underflow
```
This branch already has the related fix SPARK-60118, so backporting this
keeps decimal casts consistent with master.
### Does this PR introduce _any_ user-facing change?
Yes, the same as #59335:
- Casting a string whose value is far below the target scale's ulp now
returns the rounded value (e.g. 0.00) instead of throwing
`ArithmeticException`, and returns quickly.
- With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting a string
with a huge exponent returns NULL (non-ANSI, try_cast) or throws
`NUMERIC_VALUE_OUT_OF_RANGE` (ANSI) instead of `ArithmeticException:
Underflow`, and a decimal stored as a long no longer throws
`ArrayIndexOutOfBoundsException` for a scale change beyond the Int range. In
that error, a value whose scale is below -1000 is shown in scientific notation.
No input that previously produced a value changes its result.
### How was this patch tested?
The tests from #59335 in `DecimalSuite`, `CastSuiteBase`,
`CastWithAnsiOnSuite` and `CastWithAnsiOffSuite` (also run through
`TryCastSuite`). On this branch with JDK 17, {'4.3': '512 tests across
`DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`,
`DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`,
plus `UdfSerializationSuite` (16 tests)', '4.2': '410 tests across
`DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`,
`DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`',
'4.1': '403 tests across `DecimalSuite`, `CastWithAnsiOnSuite`,
`CastWithAnsiOffSuite`, `TryCastSuite`, `DecimalPrecisionSuite`,
`DecimalExpressionSuite` and `MathExpressionsSuite`', '4.0': '354 tests across
`DecimalSuite`, `CastWithAnsiOnSuite`, `CastWithAnsiOffSuite`, `TryCastSuite`,
`DecimalPrecisionSuite`, `DecimalExpressionSuite` and `MathExpressionsSuite`'}
pass, and scalastyle (`sql-api`, `catalyst`, `catal
yst` tests) and the `sql/api` scalafmt check pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
