This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-10509-c5e390031a1443f97355ed80411d6885e14642ff in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
commit cb3bec2362d8a1fb98ca1114f6bd0626ce8ec428 Author: David López <[email protected]> AuthorDate: Tue Oct 6 05:06:52 2026 +0000 fix(arrow-cast): round decimal to float conversions once (#10509) # Which issue does this PR close? - Closes #10508. # Rationale for this change `cast(decimal, Float64)` and `cast(decimal, Float32)` could return a float that is not the nearest one to the decimal's exact value — see the issue for the full analysis and the failing values. The short version is two independent double roundings. `single_decimal_to_float_lossy` computed `f(x) / 10_f64.powi(scale)`, where `f(x)` rounds the unscaled integer once it exceeds 2^53 and `10_f64.powi(scale)` is itself 1 ULP off for some scales a `DECIMAL(38,s)` can declare; dividing two wrongly rounded operands does not give the correctly rounded quotient. Separately, the `Float32` target computed the `f64` result and narrowed it with `as f32`, rounding a second time — a decimal just above an `f32` midpoint can collapse onto that midpoint in `f64`, and round-half-even then sends it the wrong way, so fixing the `f64` path alone leaves those cases unchanged. The expected result is the float nearest the decimal's exact value: what parsing the decimal's own text gives, and what Java's `BigDecimal.doubleValue()` / `floatValue()` give. # What changes are included in this PR? - `single_decimal_to_float_lossy` keeps the existing arithmetic as a fast path, but only where both operands are exact: `-22 <= scale <= 22` and `|unscaled| < 2^53`. A negative scale multiplies by `10^-scale` rather than dividing by `10^scale`, since only the former is an exact power of ten. Everything outside that range goes through a correctly rounded decimal-string parse. - New `single_decimal_to_f32_lossy`, used by the `Float32` target, narrowing in one step. Same shape, smaller exact range: `10^k` is exact in an `f32` only up to `k = 10` (`5^10` is the largest power of five that fits the 24-bit significand), and the integer bound is 2^24. - `Float16` is unchanged and now consumes the corrected `f64`. It is still not correctly rounded — `half::f16::from_str` is `f32::from_str(..).map(f16::from_f32)`, so there is no correctly rounded decimal → `f16` conversion to compare against. Noted as a remaining limitation. # Are these changes tested? Yes, six tests in `arrow-cast/src/cast/mod.rs`: - `Float64` and `Float32` against expected constants, each including control cases that already passed before this change. - `Decimal256`, where the `i256 -> f64` step is least likely to leave the value intact. - A test that every value lands on the same float as parsing its own text — that is the definition of correctly rounded, and it needs no expected constants. It sweeps both sides of each fast-path boundary: the 2^53 / 2^24 significand limits and the scale ±22 / ±10 limits on exact powers of ten. - A test for values carrying more digits than their declared precision, which must not be truncated. - A test for the narrow-precision path, which every other test here misses by declaring precision 38 or 76. Also checked out of tree against randomized sweeps of `(unscaled, scale)` pairs, comparing every result to an independently built decimal string, for `Decimal128` to both `f32` and `f64`: 0 mismatches, where the same sweep failed on a large share of the wide cases before the change. # Are there any user-facing changes? Yes. Casts from decimal to `Float32`/`Float64` can now return a different — correctly rounded — value for wide decimals. No public API is added or changed. **Performance** (benchmark bot): `cast decimal128 to float64` ~12% slower, `cast decimal256 to float64` ~7% faster. The cost is the per-value exactness test; the gain is the divisor being computed once per array instead of per value. # AI disclosure This change was developed with AI assistance (Claude). --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> --- arrow-cast/src/cast/mod.rs | 355 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 347 insertions(+), 8 deletions(-) diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs index f008bbcae4..0d8400a8e1 100644 --- a/arrow-cast/src/cast/mod.rs +++ b/arrow-cast/src/cast/mod.rs @@ -82,8 +82,16 @@ pub use decimal::parse_string_to_decimal_native; pub use decimal::{DecimalCast, rescale_decimal, single_float_to_decimal}; pub use string::cast_single_string_to_boolean_default; +/// Integers below 2^53 convert to an `f64` exactly. +const F64_EXACT_INT_LIMIT: f64 = 9_007_199_254_740_992.0; +/// Integers below 2^24 convert to an `f32` exactly. +const F32_EXACT_INT_LIMIT: f64 = 16_777_216.0; + /// Lossy conversion from decimal to float. /// +/// Returns the `f64` nearest to the decimal's exact value, rounding once, for any +/// scale that fits in an `i8` -- as every Arrow decimal's does. +/// /// Conversion is lossy and follows standard floating point semantics. Values /// that exceed the representable range become `INFINITY` or `-INFINITY` without /// returning an error. @@ -93,7 +101,127 @@ where D: DecimalType, F: Fn(D::Native) -> f64, { - f(x) / 10_f64.powi(scale) + let unscaled = f(x); + // Both operands are exact in this range, so this rounds once. A negative scale + // multiplies, since `10^-scale` is the exact power of ten there. + if (-22..=22).contains(&scale) && unscaled.abs() < F64_EXACT_INT_LIMIT { + return if scale >= 0 { + unscaled / 10_f64.powi(scale) + } else { + unscaled * 10_f64.powi(-scale) + }; + } + match i8::try_from(scale) { + Ok(scale) => decimal_to_f64_rounded_once::<D>(x, scale, unscaled), + Err(_) => unscaled / 10_f64.powi(scale), + } +} + +/// Rounds once for the values the arithmetic cannot convert exactly, by parsing +/// the decimal's own text. +#[cold] +#[inline(never)] +fn decimal_to_f64_rounded_once<D: DecimalType>(x: D::Native, scale: i8, unscaled: f64) -> f64 { + // `format_decimal` always writes a decimal literal, so the parse cannot fail. + // The fallback is there to keep that from being a panic, not because it runs. + D::format_decimal(x, u8::MAX, scale) + .parse::<f64>() + .unwrap_or_else(|_| unscaled / 10_f64.powi(scale.into())) +} + +/// As [`decimal_to_f64_rounded_once`], but narrowing to `f32` in one step. +/// Rounding to `f64` first and then to `f32` rounds twice: a decimal just above an +/// `f32` midpoint can collapse onto that midpoint in `f64`, and round-half-even +/// then sends it the wrong way. +#[cold] +#[inline(never)] +fn decimal_to_f32_rounded_once<D: DecimalType>(x: D::Native, scale: i8, unscaled: f64) -> f32 { + // Unreachable, as in [`decimal_to_f64_rounded_once`]. + D::format_decimal(x, u8::MAX, scale) + .parse::<f32>() + .unwrap_or_else(|_| unscaled as f32 / 10_f32.powi(scale.into())) +} + +/// Casts a decimal array to `Float64`, rounding each value once. +fn cast_decimal_to_f64<D, F>( + array: &dyn Array, + as_float: &F, + scale: i8, +) -> Result<ArrayRef, ArrowError> +where + D: DecimalType + ArrowPrimitiveType, + F: Fn(D::Native) -> f64, +{ + let array = array.as_primitive::<D>(); + // No power of ten outside this range is exactly representable. + if !(-22..=22).contains(&scale) { + let values = array + .unary::<_, Float64Type>(|x| decimal_to_f64_rounded_once::<D>(x, scale, as_float(x))); + return Ok(Arc::new(values)); + } + let pow = 10_f64.powi(scale.unsigned_abs().into()); + let values = if scale >= 0 { + array.unary::<_, Float64Type>(|x| { + let unscaled = as_float(x); + if unscaled.abs() < F64_EXACT_INT_LIMIT { + unscaled / pow + } else { + decimal_to_f64_rounded_once::<D>(x, scale, unscaled) + } + }) + } else { + array.unary::<_, Float64Type>(|x| { + let unscaled = as_float(x); + if unscaled.abs() < F64_EXACT_INT_LIMIT { + unscaled * pow + } else { + decimal_to_f64_rounded_once::<D>(x, scale, unscaled) + } + }) + }; + Ok(Arc::new(values)) +} + +/// Casts a decimal array to `Float32`, rounding each value once. Same shape as +/// [`cast_decimal_to_f64`], with the smaller bounds an `f32` allows: integers are +/// exact below 2^24, and `10^k` only up to `k = 10`. +fn cast_decimal_to_f32<D, F>( + array: &dyn Array, + as_float: &F, + scale: i8, +) -> Result<ArrayRef, ArrowError> +where + D: DecimalType + ArrowPrimitiveType, + F: Fn(D::Native) -> f64, +{ + let array = array.as_primitive::<D>(); + // No power of ten outside this range is exactly representable. + if !(-10..=10).contains(&scale) { + let values = array + .unary::<_, Float32Type>(|x| decimal_to_f32_rounded_once::<D>(x, scale, as_float(x))); + return Ok(Arc::new(values)); + } + let pow = 10_f32.powi(scale.unsigned_abs().into()); + let values = if scale >= 0 { + array.unary::<_, Float32Type>(|x| { + let unscaled = as_float(x); + if unscaled.abs() < F32_EXACT_INT_LIMIT { + unscaled as f32 / pow + } else { + decimal_to_f32_rounded_once::<D>(x, scale, unscaled) + } + }) + } else { + array.unary::<_, Float32Type>(|x| { + let unscaled = as_float(x); + if unscaled.abs() < F32_EXACT_INT_LIMIT { + unscaled as f32 * pow + } else { + decimal_to_f32_rounded_once::<D>(x, scale, unscaled) + } + }) + }; + Ok(Arc::new(values)) } /// CastOptions provides a way to override the default cast behaviors @@ -2378,13 +2506,8 @@ where <i32 as From<i8>>::from(*scale), )) }), - Float32 => cast_decimal_to_float::<D, Float32Type, _>(array, |x| { - single_decimal_to_float_lossy::<D, F>(&as_float, x, <i32 as From<i8>>::from(*scale)) - as f32 - }), - Float64 => cast_decimal_to_float::<D, Float64Type, _>(array, |x| { - single_decimal_to_float_lossy::<D, F>(&as_float, x, <i32 as From<i8>>::from(*scale)) - }), + Float32 => cast_decimal_to_f32::<D, F>(array, &as_float, *scale), + Float64 => cast_decimal_to_f64::<D, F>(array, &as_float, *scale), Utf8View => value_to_string_view(array, cast_options), Utf8 => value_to_string::<i32>(array, cast_options), LargeUtf8 => value_to_string::<i64>(array, cast_options), @@ -15288,4 +15411,220 @@ mod tests { assert_eq!(expected, actual); } + + fn decimal128(unscaled: i128, precision: u8, scale: i8) -> Decimal128Array { + Decimal128Array::from(vec![unscaled]) + .with_precision_and_scale(precision, scale) + .unwrap() + } + + #[test] + fn test_cast_decimal_to_float_is_correctly_rounded() { + // (unscaled, scale, nearest double). The last four already passed. + for (unscaled, scale, expected) in [ + (12345678901234567890i128, 2i8, 1.2345678901234568e17f64), + (10i128.pow(37), 37, 1.0), + (123456789012345678901, 20, 1.2345678901234567), + (1, 37, 1e-37), + (1, 1, 0.1), + (15, 1, 1.5), + (123456, 3, 123.456), + (-123456, 3, -123.456), + ] { + let out = cast(&decimal128(unscaled, 38, scale), &DataType::Float64).unwrap(); + assert_eq!( + out.as_primitive::<Float64Type>().value(0), + expected, + "Decimal128({unscaled}, scale={scale}) -> Float64" + ); + } + + // (unscaled, scale, nearest float). The first three differ from + // `(f32) (f64) value`, so fixing the f64 path alone leaves them wrong. + for (unscaled, scale, expected) in [ + (13631072500000000514758830i128, 18i8, 13631073.0f32), + (72073620000000000000000582908005, 24, 72073624.0), + (-3273316900000000000957536840, 20, -32733170.0), + (123456, 3, 123.456), + (-12345678, 3, -12345.678), + (1, 10, 1e-10), + ] { + let out = cast(&decimal128(unscaled, 38, scale), &DataType::Float32).unwrap(); + assert_eq!( + out.as_primitive::<Float32Type>().value(0), + expected, + "Decimal128({unscaled}, scale={scale}) -> Float32" + ); + } + } + + // Both sides of every boundary the conversion switches on. + #[test] + fn test_cast_decimal_to_float_matches_parsing_its_own_text() { + for (unscaled, precision, scale) in [ + (9007199254740991i128, 38u8, 6i8), // last integer exact in an f64 + (9007199254740992, 38, 6), // 2^53, the first that is not + (9007199254740993, 38, 6), // one past that + (16777215, 38, 3), // last integer exact in an f32 + (16777216, 38, 3), // 2^24, the first that is not + (16777217, 38, 3), // one past that + (123456789, 38, 10), // last exact power of ten for an f32 + (123456789, 38, 11), // one past it + (123456789, 38, 22), // last exact power of ten for an f64 + (123456789, 38, 23), // one past it + (12345, 38, 0), // lowest scale the division handles + (12345, 38, -1), // one below it: negative scales + (12345, 38, -23), // are all rescaled as text + // only the upper bound of a scale is validated, so this is legal + (i128::MAX, 38, i8::MIN), + (999999999999999, 15, 3), // widest value precision 15 allows + (-999999999999999, 15, 15), + (999999999999999, 16, 3), // one precision past the skipped test + (9999999, 7, 3), // widest value precision 7 allows + (-9999999, 7, 7), + (9999999, 8, 3), // one precision past it, for `f32` + (-99999999999999999999999999999999999999, 38, 20), + (1, 38, 38), + ] { + let text = Decimal128Type::format_decimal(unscaled, u8::MAX, scale); + let array = decimal128(unscaled, precision, scale); + let label = format!("Decimal128({unscaled}, precision={precision}, scale={scale})"); + + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!( + as_f64.as_primitive::<Float64Type>().value(0), + text.parse::<f64>().unwrap(), + "{label} -> Float64 disagrees with parsing {text:?}" + ); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!( + as_f32.as_primitive::<Float32Type>().value(0), + text.parse::<f32>().unwrap(), + "{label} -> Float32 disagrees with parsing {text:?}" + ); + } + } + + #[test] + fn test_cast_decimal256_to_float_is_correctly_rounded() { + for (unscaled, scale) in [ + ("12345678901234567890", 2i8), + ("10000000000000000000000000000000000000", 37), + ("13631072500000000514758830", 18), + ("123456", 3), + ] { + let value = i256::from_string(unscaled).unwrap(); + let array = Decimal256Array::from(vec![value]) + .with_precision_and_scale(76, scale) + .unwrap(); + let text = Decimal256Type::format_decimal(value, u8::MAX, scale); + + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!( + as_f64.as_primitive::<Float64Type>().value(0), + text.parse::<f64>().unwrap(), + "Decimal256({unscaled}, scale={scale}) -> Float64" + ); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!( + as_f32.as_primitive::<Float32Type>().value(0), + text.parse::<f32>().unwrap(), + "Decimal256({unscaled}, scale={scale}) -> Float32" + ); + } + } + + #[test] + fn test_cast_decimal_to_float_saturates_out_of_range() { + // 1.7e39: past f32::MAX, still far inside f64. + for (unscaled, want_f32) in [(i128::MAX, f32::INFINITY), (i128::MIN, f32::NEG_INFINITY)] { + let array = decimal128(unscaled, 38, -1); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!(as_f32.as_primitive::<Float32Type>().value(0), want_f32); + + let text = Decimal128Type::format_decimal(unscaled, u8::MAX, -1); + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!( + as_f64.as_primitive::<Float64Type>().value(0), + text.parse::<f64>().unwrap() + ); + } + + // 1e-76: below the smallest f32 subnormal, representable in an f64. + let array = Decimal256Array::from(vec![i256::from(1)]) + .with_precision_and_scale(76, 76) + .unwrap(); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!(as_f32.as_primitive::<Float32Type>().value(0), 0.0); + + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!(as_f64.as_primitive::<Float64Type>().value(0), 1e-76); + } + + #[test] + fn test_single_decimal_to_float_lossy_scale_outside_i8() { + // `format_decimal` takes an `i8` scale; narrowing 128 into one wraps it to + // -128 and turns 1e-128 into 1e128. The edges of the `i8` range still round + // once, and anything past them falls back to the division. + let as_float = |x: i128| x as f64; + let lossy = + |scale: i32| single_decimal_to_float_lossy::<Decimal128Type, _>(&as_float, 1, scale); + for scale in [i8::MAX, i8::MIN] { + let text = Decimal128Type::format_decimal(1, u8::MAX, scale); + assert_eq!( + lossy(scale.into()), + text.parse::<f64>().unwrap(), + "scale={scale}" + ); + } + for scale in [128, -129] { + assert_eq!(lossy(scale), 1.0 / 10_f64.powi(scale), "scale={scale}"); + } + } + + // Keeping only `MAX_PRECISION` digits would scale the result by ten. + #[test] + fn test_cast_decimal_to_float_keeps_digits_beyond_declared_precision() { + // 39 digits, one more than `Decimal128Type::MAX_PRECISION`. + let array = decimal128(i128::MAX, 38, 0); + + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!( + as_f64.as_primitive::<Float64Type>().value(0), + 1.7014118346046923e38 // 2^127, the double nearest i128::MAX + ); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!( + as_f32.as_primitive::<Float32Type>().value(0), + 1.7014118e38 // 2^127, the float nearest i128::MAX + ); + + // A precision of 15 (7 for `f32`) promises an integer the float holds + // exactly, but values are not validated against the precision they declare. + for (unscaled, precision, scale) in + [(12345678901234567890i128, 15u8, 2i8), (16777217, 7, 1)] + { + let text = Decimal128Type::format_decimal(unscaled, u8::MAX, scale); + let array = decimal128(unscaled, precision, scale); + + let as_f64 = cast(&array, &DataType::Float64).unwrap(); + assert_eq!( + as_f64.as_primitive::<Float64Type>().value(0), + text.parse::<f64>().unwrap(), + "{text} at precision {precision} -> Float64" + ); + + let as_f32 = cast(&array, &DataType::Float32).unwrap(); + assert_eq!( + as_f32.as_primitive::<Float32Type>().value(0), + text.parse::<f32>().unwrap(), + "{text} at precision {precision} -> Float32" + ); + } + } }
