rdtr opened a new pull request, #13642:
URL: https://github.com/apache/gluten/pull/13642

   ## What changes are proposed in this pull request?
   
   On Spark 3.4 and 3.5, `Encode` calls `String.getBytes(charset)`. Characters 
the charset cannot represent become `?`, and any JDK charset name or alias is 
accepted. Since the Velox pin moved to dft-2026_11_05 (#13540), Gluten offloads 
`Encode` to Velox `encode` (facebookincubator/velox#18534), which throws 
`encode: input contains a character that cannot be encoded using 'US-ASCII'` 
unless the query config `spark.legacy_coding_error_action` is set, and accepts 
only seven charset names. `encode('日本語', 'US-ASCII')` and `encode('café', 
'utf8')` return values in Spark and fail in Gluten.
   
   This sets `spark.legacy_coding_error_action=true` for the Velox backend on 
Spark 3.x. `GlutenConfig.getNativeSessionConf` adds it as 
`spark.gluten.sql.columnar.backend.velox.spark.legacy_coding_error_action`, 
which `overwriteVeloxConf` passes to the Velox query config, so no C++ change 
is needed. It is added before the user's backend configs are copied, so an 
explicit setting still wins.
   
   `VeloxSparkPlanExecApi.extraExpressionConverter` now falls back when the 
charset is not a string literal naming one of `US-ASCII`, `ISO-8859-1`, 
`UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16` or `UTF-32` (ASCII case-insensitive, 
like Velox), or a NULL literal. A non-literal charset could hold any alias, so 
it falls back too. The fallback reason is the new 
`EncodeRestrictions.ONLY_STANDARD_CHARSET_LITERAL`.
   
   Spark 4.x is unchanged. There `Encode` is replaced by a `StaticInvoke` that 
Gluten does not offload, so the config is not set.
   
   Fixes #13641
   
   ## How was this patch tested?
   
   New test `encode` in `ScalarFunctionsValidateSuite`, run on Spark 3.5 and 
earlier. Input is a Parquet table with ASCII, Latin-1, CJK, a supplementary 
character, an empty string and NULL.
   
   - For each of the seven charsets (in mixed case), the query is offloaded 
(`ProjectExecTransformer`) and the result equals vanilla Spark.
   - `encode(s, 'utf8')`, `encode(s, 'latin1')` and `encode(s, cs)` with a 
column charset fall back, and the fallback reason contains the new restriction 
message.
   
   Ran `ScalarFunctionsValidateSuite` on Spark 3.5: 91 succeeded. Without the 
config, the test fails with `encode: input contains a character that cannot be 
encoded using 'US-ASCII'`. Without the charset check, it fails with `encode: 
unsupported charset 'utf8'`. `test-compile` also passes on Spark 4.1 with Scala 
2.13.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to