snmvaughan opened a new issue, #7581:
URL: https://github.com/apache/datafusion-comet/issues/7581
### Describe the bug
The native string to decimal cast parses the digits of a string into an
`i128`, so a string with more than 38 digits after its leading zeros, or with
more than 38 fractional digits, fails to parse. `java.math.BigDecimal`, which
Spark parses with, has no such limit. Comet returns NULL, or raises
`CAST_INVALID_INPUT` under ANSI mode, where Spark returns a value or a
different error.
| Input, cast to the target type | Spark 4.1.3 | Comet |
|---|---|---|
| `'1.'` and 50 zeros, to `DECIMAL(10,2)` | `1.00` | NULL
(`CAST_INVALID_INPUT` under ANSI) |
| `'0.'`, 40 zeros and `'1'`, to `DECIMAL(10,2)` | `0.00` | NULL
(`CAST_INVALID_INPUT` under ANSI) |
| `'1'` and 39 zeros, to `DECIMAL(10,-35)`, with the setting below |
`1.0000E+39` | NULL (`CAST_INVALID_INPUT` under ANSI) |
| `'1'` and 39 zeros, to `DECIMAL(10,2)`, with the setting below | NULL
(`NUMERIC_VALUE_OUT_OF_RANGE` under ANSI) | NULL (`CAST_INVALID_INPUT` under
ANSI) |
The last two rows need `spark.sql.legacy.allowNegativeScaleOfDecimal=true`.
Without it, Spark rejects the 40-digit string before it rescales it: NULL, or
`NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI. The first two rows need no setting.
### Steps to reproduce
```sql
SET spark.sql.ansi.enabled=false;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES
('1.00000000000000000000000000000000000000000000000000');
SELECT CAST(s AS DECIMAL(10, 2)) FROM t;
-- Spark: 1.00
-- Comet: NULL
```
The value comes from a table so that the optimizer doesn't fold the cast
before Comet sees it. With `spark.sql.ansi.enabled=true`, Comet raises
`CAST_INVALID_INPUT` and Spark returns `1.00`.
### Expected behavior
The native cast returns what Spark returns: the rounded value for the first
two rows, `1.0000E+39` for the negative-scale target, and
`NUMERIC_VALUE_OUT_OF_RANGE` under ANSI where the value is too large for the
target type.
### Additional context
- The limit is in `parse_decimal_str` in
`native/spark-expr/src/conversion_funcs/string.rs`. `digits_to_i128` fails on
the integral or the fractional digits, and `pow10_i128(fractional_scale)` has
no entry above 10^38. Both map to `invalid_decimal_cast`.
- Spark rounds `HALF_UP`, so only the digits up to the target scale and the
first digit after it decide the result. A fix can work on the digit string and
does not need an integer type wider than `i128`.
- Related: #7576 is the same function reporting the wrong error for a scale
adjustment over 38, which is a one-line change. #7188 and #7578 make a string
with more than 38 integral digits raise `NUMERIC_OUT_OF_SUPPORTED_RANGE`, as
Spark does, when the setting is off.
### Willingness to contribute
I can contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
