andygrove opened a new pull request, #5080: URL: https://github.com/apache/datafusion-comet/pull/5080
## Which issue does this PR close? Closes #5066. ## Rationale for this change `MakeDecimal` with `nullOnOverflow = false` (ANSI mode) must fail on overflow, matching Spark's `Decimal.set(unscaled, precision, scale)`, which throws `NUMERIC_VALUE_OUT_OF_RANGE`. The Scala serde already serializes `failOnError = !nullOnOverflow`, but the native registration of `make_decimal` discarded the flag, so Comet returned NULL where Spark throws. Since `MakeDecimal` is generated by the `DecimalAggregates` optimizer rule, the divergence surfaces on decimal aggregations rather than user-written SQL. ## What changes are included in this PR? - `spark_make_decimal` now takes `fail_on_error` and raises `SparkError::NumericValueOutOfRange` (via the existing `decimal_overflow_error` helper, as `CheckOverflow` already does) when the unscaled long does not fit the target precision. Non-ANSI behavior is unchanged: overflow becomes NULL. - The `make_decimal` registration in `comet_scalar_funcs.rs` threads `fail_on_error` through instead of dropping it. - Test-only: `createMakeDecimalColumn` in `ShimCometTestBase` (all Spark version variants) gained an overload that accepts `nullOnOverflow`, since the existing helper hardcoded the null-on-overflow behavior. ## How are these changes tested? - New Rust unit tests in `make_decimal.rs` covering array and scalar overflow with `fail_on_error` set (error) and unset (NULL), plus NULL input under `fail_on_error`. - New JVM test `make decimal using DataFrame API - overflow throws when nullOnOverflow=false` in `CometExpressionSuite`, asserting Spark and Comet both fail with the `cannot be represented as Decimal(3, 0)` message. The test was verified to fail before the native fix and pass after it. This issue was found by an audit run with the `audit-comet-expression` project skill (tracked under epic #5078). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
