yongster commented on code in PR #10707:
URL: https://github.com/apache/arrow-rs/pull/10707#discussion_r3804155621
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -348,6 +348,20 @@ pub fn cast(array: &dyn Array, to_type: &DataType) ->
Result<ArrayRef, ArrowErro
cast_with_options(array, to_type, &CastOptions::default())
}
+/// Convert an integer to a decimal native value without wrapping.
+///
+/// `AsPrimitive` / `as` silently truncates when the source is wider than `M`
+/// (for example `5_000_000_000i64 as i32`). All integer sources fit in `i128`
+/// losslessly, which [`DecimalCast`] then converts to the decimal native type
+/// with a range check.
Review Comment:
Applied, thank you. I also noted that for always-fitting conversions this
should optimize down to a plain integer widening.
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -360,43 +374,62 @@ fn cast_integer_to_decimal<
cast_options: &CastOptions,
) -> Result<ArrayRef, ArrowError>
where
- <T as ArrowPrimitiveType>::Native: AsPrimitive<M>,
- M: ArrowNativeTypeOp,
+ <T as ArrowPrimitiveType>::Native: ArrowNativeTypeOp + Into<i128>,
+ M: ArrowNativeTypeOp + DecimalCast,
{
- let scale_factor = base.pow_checked(scale.unsigned_abs() as
u32).map_err(|_| {
+ let overflow = |v: T::Native| {
ArrowError::CastError(format!(
- "Cannot cast to {:?}({}, {}). The scale causes overflow.",
+ "Cannot cast to {}({precision}, {scale}). Overflowing on {v:?}",
D::PREFIX,
- precision,
- scale,
))
- })?;
+ };
let array = if scale < 0 {
- match cast_options.safe {
- true => array.unary_opt::<_, D>(|v| {
- v.as_()
- .div_checked(scale_factor)
+ // Compute the scale factor once in the source type. Scaling before the
+ // checked conversion permits values that only fit the decimal native
+ // type after scaling.
+ let scale_factor = T::Native::usize_as(10)
+ .pow_checked(scale.unsigned_abs() as u32)
+ .ok();
+
+ match (scale_factor, cast_options.safe) {
+ (Some(scale_factor), true) => array.unary_opt::<_, D>(|v| {
+ v.div_checked(scale_factor)
.ok()
+ .and_then(integer_to_decimal_native::<_, M>)
.and_then(|v| (D::is_valid_decimal_precision(v,
precision)).then_some(v))
}),
- false => array.try_unary::<_, D, _>(|v| {
- v.as_()
- .div_checked(scale_factor)
+ (Some(scale_factor), false) => array.try_unary::<_, D, _>(|v| {
+ v.div_checked(scale_factor)
+ .ok()
+ .and_then(integer_to_decimal_native::<_, M>)
+ .ok_or_else(|| overflow(v))
.and_then(|v| D::validate_decimal_precision(v, precision,
scale).map(|()| v))
})?,
+ // A scale factor that overflows the source type is larger than all
+ // source values, so integer division produces zero.
Review Comment:
Applied, thanks. I added the note that this overflow path should be
unreachable for a well formed decimal scale.
##########
arrow-cast/src/cast/mod.rs:
##########
@@ -10622,6 +10647,121 @@ mod tests {
assert!(casted_array.is_err());
}
+ #[test]
+ fn test_cast_integer_to_decimal32_does_not_truncate() {
+ let array = Int64Array::from(vec![5_000_000_000i64, 10_000_000_000,
42]);
+ let safe = CastOptions {
+ safe: true,
+ format_options: FormatOptions::default(),
+ };
+ let unsafe_opts = CastOptions {
+ safe: false,
+ format_options: FormatOptions::default(),
+ };
+
+ let result = cast_with_options(&array, &DataType::Decimal32(9, 0),
&safe).unwrap();
+ let result = result.as_primitive::<Decimal32Type>();
+ assert!(
+ result.is_null(0),
+ "5e9 must not wrap to {}",
+ result.value(0)
+ );
+ assert!(result.is_null(1));
+ assert_eq!(result.value(2), 42);
+
+ let err = cast_with_options(&array, &DataType::Decimal32(9, 0),
&unsafe_opts)
+ .unwrap_err()
+ .to_string();
+ assert!(
+ err.contains("5000000000"),
+ "unsafe error should report the original value, got {err}"
+ );
+ assert!(
+ !err.contains("705032704"),
Review Comment:
Good point. The tests now assert the full error string, e.g.
`Cast error: Cannot cast to Decimal32(9, 0). Overflowing on 5000000000`
so it is clear we report the original input rather than a truncated value.
##########
arrow/benches/cast_kernels.rs:
##########
@@ -238,6 +238,8 @@ fn cast_array(array: &ArrayRef, to_type: DataType) {
fn add_benchmark(c: &mut Criterion) {
let i32_array = build_array::<Int32Type>(512);
let i64_array = build_array::<Int64Type>(512);
+ let i64_decimal32_array: ArrayRef =
Arc::new(Int64Array::from_iter_values(0..512));
+ let i64_decimal32_scaled_array: ArrayRef =
Arc::new(Int64Array::from_value(5_000_000_000, 512));
Review Comment:
Good call. The extra arrays are gone; both benches now reuse the existing
`i64_array`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
