rdtr opened a new pull request, #13598: URL: https://github.com/apache/gluten/pull/13598
## What changes are proposed in this pull request? Spark rewrites `try_add` / `try_subtract` on intervals to `TryEval(Add(left, right, EvalMode.ANSI))` and `TryEval(Subtract(...))` (`TryEval.scala:85` and `:199` on branch-4.2). Interval addition and subtraction always use `IntervalMathUtils.addExact` / `subtractExact` (`arithmetic.scala:455-456` and `:546-547`), so plain `a + b` throws on overflow in both ANSI modes and `try_add(a, b)` returns NULL. Velox has the matching functions since facebookincubator/velox#17098: `add` / `subtract` on `INTERVAL YEAR TO MONTH` throw "Arithmetic overflow", and `try()` turns that into NULL. Gluten fell back on both forms. This makes two changes: - `ConverterUtils.getTypeSigName` maps `YearMonthIntervalType.DEFAULT` to `iyear`. Without it, any scalar function with a year-month interval argument failed validation with `Type YearMonthIntervalType(0,1) not supported.`, including plain `a + b`. The native side only reads the argument names of scalar function signatures as strings, so no C++ change is needed. Aggregate signatures are parsed into Velox types and do not know `iyear` yet, so an aggregate on a year-month interval still falls back, now in native validation. Other scalar functions that Velox registers for `INTERVAL YEAR TO MONTH` are now offloaded as well, such as comparisons, `coalesce`, `hash` and unary minus, along with filters and joins on interval keys. The tests below cover that set. - `VeloxSparkPlanExecApi.genTryArithmeticTransformer` accepts year-month intervals for `Add` and `Subtract` and maps them to `try(add)` / `try(subtract)`. It uses the plain names because Velox registers the interval overloads under both `add` and `checked_add`, and the `checked_` names are being phased out. Integral types still use `checked_*`. `try_multiply` / `try_divide` on intervals still fall back, since Velox has no Spark interval multiply or divide. Day-time intervals are not covered. Gluten has no Substrait type mapping for `DayTimeIntervalType` (`ConverterUtils.getTypeNode` maps only `YearMonthIntervalType.DEFAULT`), so they keep falling back. Year-month interval types other than `YEAR TO MONTH` (for example `INTERVAL '1' YEAR`) also keep falling back, as before. The overflow error message differs. Spark throws `INTERVAL_ARITHMETIC_OVERFLOW`, while Velox throws "Arithmetic overflow". Fixes #13597 ## How was this patch tested? `MathFunctionsValidateSuite` gets two tests on a parquet table with a normal pair, a pair whose sum overflows the int month count, a pair whose difference overflows, and a NULL row. - "year-month interval add and subtract, with and without try": `try_add` / `try_subtract` results equal vanilla Spark, the plan contains `ProjectExecTransformer`, and the overflow rows are NULL. `a + b` / `a - b` on the non-overflowing rows equal vanilla Spark and are offloaded. `a + b` on the overflowing row throws. - "year-month interval comparisons, coalesce, hash, negation, filter and join": `a > b`, `a = b`, `a < b`, `coalesce(a, b)`, `hash(a)` and `-b` in a `ProjectExecTransformer`, `WHERE a > b` in a `FilterExecTransformer`, and a self-join on `a` in a hash join transformer, each equal to vanilla Spark. Both pass on Spark 3.5 and Spark 4.2. With the signature name reverted, both tests fail with `Type YearMonthIntervalType(0,1) not supported.` With only the `genTryArithmeticTransformer` change reverted, the first test fails with `try is not supported`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
