snmvaughan opened a new issue, #7576:
URL: https://github.com/apache/datafusion-comet/issues/7576
### Describe the bug
Under ANSI mode, a string whose integral part has up to 38 digits but whose
value does not fit the target decimal type raises
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` in Spark. The native cast raises
`CAST_INVALID_INPUT`, which calls the value malformed, when the difference
between the target scale and the parsed scale is more than 38.
`parse_string_to_decimal` returns `Ok(None)` for that case ("scale is too high
to fit i128"), and the cast reports `Ok(None)` as an invalid input under ANSI.
Outside ANSI both engines return NULL, so only the error differs.
| Input, cast under ANSI | Spark 4.1.3 | Comet |
|---|---|---|
| `'1e37'` to `DECIMAL(10,2)` | `NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION`
| `CAST_INVALID_INPUT` |
| `'1e30'` to `DECIMAL(38,10)` |
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` | `CAST_INVALID_INPUT` |
| `'1e29'` to `DECIMAL(38,10)` |
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` | `CAST_INVALID_INPUT` |
| `'1e30'` to `DECIMAL(10,2)` (scale adjustment 32, for comparison) |
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` |
`NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` |
### Steps to reproduce
```sql
SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING) USING parquet;
INSERT INTO t VALUES ('1e37');
SELECT CAST(s AS DECIMAL(10, 2)) FROM t;
-- Spark: [NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 1E+37 cannot be
represented as Decimal(10, 2). ...
-- Comet: [CAST_INVALID_INPUT] The value '1e37' of the type "STRING" cannot
be cast to "DECIMAL(10,2)" because it is malformed. ...
```
The value comes from a table so that the optimizer doesn't fold the cast
before Comet sees it.
### Expected behavior
The native cast raises `NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION` for a
string that parses but does not fit the target type, as Spark does.
### Additional context
- The cause is the `return Ok(None)` for `scale_adjustment > 38` in
`parse_string_to_decimal`. Returning the same `NumericValueOutOfRange` error as
the other overflow paths would match Spark, and NULL outside ANSI is unchanged.
- It is separate from the 38-integral-digit limit in #7188, where Spark
raises `NUMERIC_OUT_OF_SUPPORTED_RANGE` before it looks at the target type.
These strings have up to 38 integral digits.
### Willingness to contribute
I can contribute a fix for this bug independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
