peterxcli opened a new issue, #5670:
URL: https://github.com/apache/datafusion-comet/issues/5670
### Describe the bug
`CAST(decimal AS DOUBLE)` / `CAST(decimal AS FLOAT)` is delegated to
arrow-cast, which computes `(unscaled as f64) / 10^scale` — two roundings for
DOUBLE and three for FLOAT (via an f64 intermediate). Spark uses
`BigDecimal.doubleValue` / `floatValue`, which round the exact decimal once.
Whenever `|unscaled| >= 2^53` and `scale >= 1` the two can differ in the last
ulp; for `DECIMAL(38,18)` that is essentially every `|value| >= 0.009` (about
26% of random values differ in a model of arrow's formula). The wrong double is
visible in string output, equality filters/joins on the cast value, and sums.
Location: `native/spark-expr/src/conversion_funcs/numeric.rs:64-85` —
`is_df_cast_from_decimal_spark_compatible` lists `Float32`/`Float64`, so
`cast.rs:402-407` falls through to arrow's cast. Affects LEGACY, ANSI and TRY
alike.
### Steps to reproduce
```scala
val p = "/tmp/dec_to_double"
Seq("12345.6789", "123456.789012", "76543.21").toDF("s")
.selectExpr("CAST(s AS DECIMAL(38,18)) AS
c").write.mode("overwrite").parquet(p)
spark.read.parquet(p).selectExpr("CAST(c AS DOUBLE)").show(false)
```
| value | Spark | Comet |
|---|---|---|
| 12345.6789 | 12345.6789 | 12345.678899999999 |
| 123456.789012 | 123456.789012 | 123456.78901200001 |
| 76543.21 | 76543.21 | 76543.20999999999 |
FLOAT: `SELECT CAST(CAST('16777217.0000000001' AS DECIMAL(38,10)) AS FLOAT)`
→ Spark `1.6777218E7`, Comet `1.6777216E7` (the f64 intermediate is exactly
16777217.0, then ties-to-even).
### Expected behavior
Match Spark, i.e. a correctly rounded conversion of the exact decimal value
(`BigDecimal.doubleValue` / `floatValue`).
### Proposed solution
Remove `Float32`/`Float64` from `is_df_cast_from_decimal_spark_compatible`
and add a correctly rounded `Decimal128 -> f64/f32` kernel in `numeric.rs`:
- fast path when `|unscaled| < 2^53` and `scale <= 22`: `unscaled as f64 /
10^scale` is then a single correctly rounded operation (exactly Java's fast
path);
- otherwise format `<unscaled>e-<scale>` into a stack buffer and parse with
Rust's correctly rounded `f64::from_str`; for FLOAT parse the same string with
`f32::from_str` directly (never through f64).
Add `DECIMAL(38,18)` values such as `12345.6789` and `76543.21` to
`CometNativeCastSuite`; the existing DECIMAL(38,18) cases only use values whose
double happens to coincide, which is why CI is green.
### Additional context
PR #4278 excluded DECIMAL(38,18) from the nested cast matrix citing "the
same known decimal-to-floating ULP mismatch", but no issue was filed. #3948
added the DECIMAL(38,18) → Float/Double scalar tests with non-diverging values.
#286 listed "Decimal to Floating-point — seems correct but needs tests to
confirm".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
