rdtr opened a new pull request, #13529: URL: https://github.com/apache/gluten/pull/13529
## What changes are proposed in this pull request? Spark resolves ANSI mode per expression during analysis and stores it in the expression (`failOnError`, `ansiEnabled` or `EvalMode`). Normally that is the session's mode, but a view runs its body with the mode it was created with (SPARK-33141, and since Spark 4.0 the ANSI setting is always captured, SPARK-53348). So after an upgrade to Spark 4, a query that reads a view created with ANSI off runs the view's expressions with ANSI off and the rest with ANSI on. Velox's Spark functions have no per-call mode. Some have a name per mode (`add` / `checked_add` / `try_add`, the cast special forms, `get`, `try_make_timestamp`): Gluten picks the name from the expression, and those are always right. The others read `spark.ansi_enabled` from the query config, which Gluten sets from the session: `abs`, `unaryminus`, `remainder`, `pmod`, integral `round` / `bround`, `conv`, `elt`, `make_date`, `make_timestamp`, `unix_timestamp`, `to_unix_timestamp`, `get_timestamp`, `next_day`, `element_at`, `sum`, `avg`, and `ceil` / `floor` on doubles. When such an expression captured the other mode, Velox would run it in the session's mode and return a different result from Spark. With `spark.gluten.sql.ansiFallback.enabled=false` this is a wrong result today, for example `abs(c)` in a view created with ANSI off wraps around in Spark but throws in Velox when the session has ANSI on. This adds one rule, `AnsiModeMismatch`, that lists the expressions whose Velox function reads the session config and compares the mode captured in the expression with the session's. On a mismatch the expression is not offloaded and the operator falls back to Spark. It is wired into `VeloxValidatorApi.doExprValidate`. Scalar expressions reach it through `ExpressionConverter`; `VeloxAggregateFunctionsBuilder`, which replaces the generic aggregate builder for the Velox backend, now calls it too, so `sum` and `avg` in a view follow the view's mode as well. Expressions with a function per mode are not listed and stay native. The same PR stops offloading `array[index]` (`GetArrayItem` with `failOnError`): Velox's `get` never throws, so with ANSI on the subscript returned NULL instead of an error. The config-reading `get_array_item` in https://github.com/facebookincubator/velox/pull/20020 will take it back. This follows the direction agreed in https://github.com/facebookincubator/velox/issues/3869: Velox functions read the session config, `try_*` and fixed-mode expressions are separate functions, and Gluten falls back when the captured mode differs from the session. ## How was this patch tested? `AnsiModeMismatchSuite` (ANSI fallback disabled): a view created with ANSI off and read with ANSI on, and the reverse, fall back for `abs` and `%` and stay native for `+`, with results compared against vanilla Spark; a view read in the mode it was created with stays native; `sum` in a view follows the view's mode; `array[index]` with ANSI on falls back while `get()` stays native. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
