rdtr opened a new pull request, #13598:
URL: https://github.com/apache/gluten/pull/13598

   ## What changes are proposed in this pull request?
   
   Spark rewrites `try_add` / `try_subtract` on intervals to `TryEval(Add(left, 
right, EvalMode.ANSI))` and `TryEval(Subtract(...))` (`TryEval.scala:85` and 
`:199` on branch-4.2). Interval addition and subtraction always use 
`IntervalMathUtils.addExact` / `subtractExact` (`arithmetic.scala:455-456` and 
`:546-547`), so plain `a + b` throws on overflow in both ANSI modes and 
`try_add(a, b)` returns NULL. Velox has the matching functions since 
facebookincubator/velox#17098: `add` / `subtract` on `INTERVAL YEAR TO MONTH` 
throw "Arithmetic overflow", and `try()` turns that into NULL. Gluten fell back 
on both forms.
   
   This makes two changes:
   
   - `ConverterUtils.getTypeSigName` maps `YearMonthIntervalType.DEFAULT` to 
`iyear`. Without it, any scalar function with a year-month interval argument 
failed validation with `Type YearMonthIntervalType(0,1) not supported.`, 
including plain `a + b`. The native side only reads the argument names of 
scalar function signatures as strings, so no C++ change is needed. Aggregate 
signatures are parsed into Velox types and do not know `iyear` yet, so an 
aggregate on a year-month interval still falls back, now in native validation. 
Other scalar functions that Velox registers for `INTERVAL YEAR TO MONTH` are 
now offloaded as well, such as comparisons, `coalesce`, `hash` and unary minus, 
along with filters and joins on interval keys. The tests below cover that set.
   - `VeloxSparkPlanExecApi.genTryArithmeticTransformer` accepts year-month 
intervals for `Add` and `Subtract` and maps them to `try(add)` / 
`try(subtract)`. It uses the plain names because Velox registers the interval 
overloads under both `add` and `checked_add`, and the `checked_` names are 
being phased out. Integral types still use `checked_*`. `try_multiply` / 
`try_divide` on intervals still fall back, since Velox has no Spark interval 
multiply or divide.
   
   Day-time intervals are not covered. Gluten has no Substrait type mapping for 
`DayTimeIntervalType` (`ConverterUtils.getTypeNode` maps only 
`YearMonthIntervalType.DEFAULT`), so they keep falling back. Year-month 
interval types other than `YEAR TO MONTH` (for example `INTERVAL '1' YEAR`) 
also keep falling back, as before.
   
   The overflow error message differs. Spark throws 
`INTERVAL_ARITHMETIC_OVERFLOW`, while Velox throws "Arithmetic overflow".
   
   Fixes #13597
   
   ## How was this patch tested?
   
   `MathFunctionsValidateSuite` gets two tests on a parquet table with a normal 
pair, a pair whose sum overflows the int month count, a pair whose difference 
overflows, and a NULL row.
   
   - "year-month interval add and subtract, with and without try": `try_add` / 
`try_subtract` results equal vanilla Spark, the plan contains 
`ProjectExecTransformer`, and the overflow rows are NULL. `a + b` / `a - b` on 
the non-overflowing rows equal vanilla Spark and are offloaded. `a + b` on the 
overflowing row throws.
   - "year-month interval comparisons, coalesce, hash, negation, filter and 
join": `a > b`, `a = b`, `a < b`, `coalesce(a, b)`, `hash(a)` and `-b` in a 
`ProjectExecTransformer`, `WHERE a > b` in a `FilterExecTransformer`, and a 
self-join on `a` in a hash join transformer, each equal to vanilla Spark.
   
   Both pass on Spark 3.5 and Spark 4.2. With the signature name reverted, both 
tests fail with `Type YearMonthIntervalType(0,1) not supported.` With only the 
`genTryArithmeticTransformer` change reverted, the first test fails with `try 
is not supported`.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-13597][VL] Offlo... rdtr (via GitHub) via commits

Reply via email to