andygrove commented on code in PR #5193:
URL: https://github.com/apache/datafusion-comet/pull/5193#discussion_r3722211086
##########
native/spark-expr/src/math_funcs/internal/make_decimal.rs:
##########
@@ -45,15 +46,34 @@ pub fn spark_make_decimal(
ColumnarValue::Array(a) => match a.data_type() {
DataType::Int64 => {
let arr = a.as_primitive::<Int64Type>();
- let mut result = Decimal128Builder::new();
- for v in arr.into_iter() {
- result.append_option(long_to_decimal(v, precision, scale,
fail_on_error)?)
- }
let result_type = DataType::Decimal128(precision, scale);
- Ok(ColumnarValue::Array(Arc::new(
- result.finish().with_data_type(result_type),
- )))
+ // The Int64 is already the unscaled Decimal128 value; this
widens the bits
+ // (an Arrow Int64->Decimal cast would rescale). Infallible so
it vectorizes.
+ let widened: Decimal128Array = unary::<_, _,
Decimal128Type>(arr, |v| v as i128);
+
+ // `.iter().flatten()` skips null slots so garbage under a
null cannot
+ // trigger a false overflow. `find` short-circuits like
`.all(is_valid)`
+ // while also handing back the value needed for the ANSI error
message.
+ let first_offender = widened
+ .iter()
+ .flatten()
+ .find(|v| !Decimal128Type::is_valid_decimal_precision(*v,
precision));
+
+ let result = match (first_offender, fail_on_error) {
+ // No overflow: attach metadata.
`with_precision_and_scale` would rescan.
Review Comment:
I'm not sure this comment is correct. I do not think
with_precision_and_scale rescans in arrow 58.4.0. It calls
validate_decimal_precision_and_scale::<T>(precision, scale), which only
bounds-checks the two numbers, and then swaps the data_type field.
Maybe the comment does not even need to mention `with_precision_and_scale`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
