minni31 opened a new pull request, #13229:
URL: https://github.com/apache/gluten/pull/13229
## What changes are proposed in this pull request?
Spark replaces `right(str, len)` with
`If(IsNull(str), null, If(len <= 0, '', Substring(str, -len, Int.MaxValue)))`
before physical planning, so Gluten runs it as native `if` and `substring`.
This PR recognizes that replacement in the Velox backend's
`extraExpressionConverter` and converts it to
`if(is_null(str), null, right(str, len))`, which calls the Spark `right`
function added to Velox in facebookincubator/velox#19377.
- The match requires the same string and the same length in both places
(`semanticEquals`). Nondeterministic expressions are never semantically
equal, so they are not converted.
- The match accepts both `UnaryMinus` forms used by supported Spark versions.
Strings with a non-default collation are not converted.
- The outer `if` keeps Spark's evaluation order: `len` is not evaluated when
`str` is null.
- If `right`, `substring`, `is_null` or `if` is in
`spark.gluten.expression.blacklist`, the expression is not converted and
runs as before.
This depends on facebookincubator/velox#19377 being in Gluten's Velox
dependency. Until then, the native validation of `right` fails and the
project falls back to Spark.
## How was this patch tested?
Added tests to `VeloxStringFunctionsSuite` that compare results with vanilla
Spark and check whether the Substrait plan calls `right`:
- `right(s, n)` and `right(s, CAST(n AS STRING))` over ASCII, multi-byte,
surrogate-pair and combining-character strings; null string and null
length; negative, zero and too-large lengths; `Int.MinValue` and
`Int.MaxValue`.
- Each of `right`, `substring`, `is_null` and `if` blacklisted.
- The exact replacement written by hand is converted. Near matches (another
threshold, another empty value, another substring length, another string,
another length) and a nondeterministic length are not.
`spotless:check` passes. The tests were not run locally; they need a Velox
build that includes facebookincubator/velox#19377.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.89 (GPT-5.6 Sol)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
