peterxcli opened a new pull request, #5684: URL: https://github.com/apache/datafusion-comet/pull/5684
## Which issue does this PR close? Closes #5670. ## Rationale for this change `CAST(decimal AS DOUBLE/FLOAT)` was delegated to arrow-cast, which computes `(unscaled as f64) / 10^scale` and rounds twice (three times for FLOAT, via the f64 intermediate). Spark's `Decimal.toDouble/toFloat` use `BigDecimal.doubleValue()/floatValue()`, which round the exact decimal once, so results differed in the last ulp whenever `|unscaled| > 2^53` — for `DECIMAL(38,18)` that is essentially every value of magnitude >= 0.01 (`12345.6789` became `12345.678899999999`; `CAST(16777217.0000000001 AS FLOAT)` gave `16777216` instead of `16777218`). ## What changes are included in this PR? - New `Decimal128 -> Float64/Float32` kernels in `numeric.rs` mirroring Java's algorithm: a single IEEE division/multiplication when the unscaled value and the power of ten are both exact in the target type (`|unscaled| <= 2^53` and `scale <= 22` for f64, `<= 2^24` / `<= 10` for f32), otherwise a correctly rounded parse of the exact decimal `<unscaled>e<-scale>` from a stack buffer straight into the target width (never via f64 for FLOAT). Negative scales and out-of-range floats (`±Infinity`) behave like `BigDecimal`. - `cast.rs` routes `(Decimal128, Float32|Float64)` to the kernels for LEGACY, ANSI and TRY (the conversion cannot fail); `Float32/Float64` are removed from `is_df_cast_from_decimal_spark_compatible`. ## How are these changes tested? - Rust unit tests: the issue's values, ties-to-even cases (`2^53+1`, `2^24+1`), 38-digit extremes, `i128::MIN/MAX`, zero, negative scales, null preservation in every eval mode, and a 30k-value randomized check that the fast paths match the correctly rounded exact path; the tests carry `assert_ne!` guards showing the old `(unscaled as f64)/10^scale` formula fails on the issue's values. Reference values were cross-checked with `jshell` (`BigDecimal.doubleValue()/floatValue()`). `cargo test -p datafusion-comet-spark-expr`: 663 passed; `cargo clippy --all-targets --workspace -- -D warnings` clean. - `CometNativeCastSuite`: `DECIMAL(38,18)` values from the issue added to the Float/Double tests, new `DECIMAL(38,10) -> Float/Double` tests (including `16777217.0000000001`), and `DecimalType(38,18)` re-enabled in the array and nested-array cast matrices that #4278 had excluded for this mismatch. Suite passes on Spark 4.1.3 (170/170, 8 pre-existing ignores). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
