andygrove opened a new issue, #5071: URL: https://github.com/apache/datafusion-comet/issues/5071
### Describe the bug For several arithmetic expressions, Comet's ANSI throw/NULL decision is correct, but the error class, error message, or exception parameters differ from Spark. All of these surface to users who catch or match on Spark error conditions. 1. **Byte/Short overflow raises the wrong error class.** `Add`/`Subtract`/`Multiply` on TINYINT/SMALLINT under ANSI: Spark 4.1 raises `BINARY_ARITHMETIC_OVERFLOW` carrying the two operand values and the operator symbol (`types/numerics.scala`, codegen in `arithmetic.scala`). Comet raises `ARITHMETIC_OVERFLOW` with `from_type: "integer"` (`native/spark-expr/src/math_funcs/checked_arithmetic.rs:69-71, 93-96`). `SparkError::BinaryArithmeticOverflow` already exists (`native/common/src/error.rs:87-93`) and is wired through `ShimSparkErrorConverter`, but no kernel constructs it. 2. **Long overflow says "integer overflow".** `from_type` is hard-coded to `"integer"` regardless of Int32 vs Int64 (`checked_arithmetic.rs:70, 94`). Spark says `"long overflow"` for BIGINT. The same applies to `SumInteger` (`native/spark-expr/src/agg_funcs/sum_int.rs`), where Spark's sum result type is always LONG. 3. **The `try_` suggestion is always empty.** `ShimSparkErrorConverter.scala:118` passes `""` for the alternative, so Comet's message omits ``Use `try_add` to tolerate overflow and return NULL instead.`` 4. **`UnaryMinus` scalar path for Byte/Short** passes `" caused"` as the type (`native/spark-expr/src/math_funcs/negative.rs:157, 160`), producing `"[ARITHMETIC_OVERFLOW] caused overflow."` instead of `"byte overflow"` / `"short overflow"`. The array path uses the correct names. 5. **`Abs`** passes Rust type names (`"Int8"`, `"Int64"`, `"Decimal128"`, ...) as `from_type` (`native/spark-expr/src/math_funcs/abs.rs:98, 106, 114, 122, 156, 186`), so Comet says `"Int64 overflow"` where Spark says `"long overflow"`. Contrast with `negative.rs`, which uses the Spark names. 6. **`avg` overflow suggestion**: `avg_decimal.rs` uses `decimal_sum_overflow_error(...)`; verify the function-name parameter produces `try_avg` (Spark's message) rather than `try_sum` (`ShimSparkErrorConverter.scala:123-127` builds ``try_$functionName``). ### Steps to reproduce ```sql SET spark.sql.ansi.enabled=true; SELECT CAST(127 AS TINYINT) + CAST(1 AS TINYINT); -- Spark: BINARY_ARITHMETIC_OVERFLOW; Comet: ARITHMETIC_OVERFLOW "integer overflow" SELECT CAST(9223372036854775807 AS LONG) + 1L; -- Spark: "long overflow"; Comet: "integer overflow" SELECT -CAST(-128 AS TINYINT); -- literal/scalar path: Comet message is "[ARITHMETIC_OVERFLOW] caused overflow." SELECT abs(CAST(-9223372036854775808 AS LONG)); -- Spark: "long overflow"; Comet: "Int64 overflow" ``` ### Expected behavior Error classes, type names, and `try_` suggestions match Spark exactly. Existing tests pass because they assert message substrings only; assertions on `getCondition()` / error class would catch these. ### Additional context Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. Possibly related to the Spark 4.2 ANSI test failures in #4967. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
