rdtr opened a new issue, #13641:
URL: https://github.com/apache/gluten/issues/13641

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   ### Bug description
   
   **[Expected behavior]** On Spark 3.4 and 3.5, `Encode` is a plain expression 
that calls `String.getBytes(charset)` (v3.5.9 
`stringExpressions.scala:2706-2709`). Characters the charset cannot represent 
become `?`, and any charset name or alias the JDK knows is accepted. 
`encode('日本語', 'US-ASCII')` returns `3F3F3F`, and `encode('café', 'utf8')` 
returns `636166C3A9`. Spark 3.x has no config to change either behavior 
(`spark.sql.legacy.javaCharsets` and `spark.sql.legacy.codingErrorAction` 
arrived in Spark 4.0).
   
   **[Actual behavior]** Since the Velox pin moved to dft-2026_11_05 (#13540), 
which brought Velox `encode` (facebookincubator/velox#18534), Gluten offloads 
`Encode` through `Sig[Encode]` (`ExpressionMappings.scala:102`). Before that 
pin Velox had no `encode` and the expression fell back. Velox `encode` throws 
on unmappable characters unless the query config 
`spark.legacy_coding_error_action` is set, and Gluten never sets it. The query 
fails with:
   
   ```
   Error Source: USER
   Error Code: INVALID_ARGUMENT
   Reason: encode: input contains a character that cannot be encoded using 
'US-ASCII'
   ```
   
   Velox also accepts only seven charset names (`US-ASCII`, `ISO-8859-1`, 
`UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16`, `UTF-32`, case-insensitive), so 
`encode(s, 'utf8')` or `encode(s, 'latin1')` fails with `encode: unsupported 
charset 'utf8'`.
   
   The Velox docs for `encode` say Spark 3.x must set 
`spark.legacy_coding_error_action=true` 
(`velox/docs/functions/spark/string.rst`). Spark 4.x is not affected: there 
`Encode` is a `RuntimeReplaceable` whose replacement is a `StaticInvoke` that 
Gluten does not offload.
   
   Repro on Spark 3.5 with the Velox backend:
   
   ```sql
   CREATE TABLE t USING parquet AS SELECT * FROM VALUES ('日本語'), ('café') AS 
v(s);
   SELECT encode(s, 'US-ASCII') FROM t;  -- fails
   SELECT encode(s, 'utf8') FROM t;      -- fails
   ```
   
   **Request**: set `spark.legacy_coding_error_action=true` on Spark 3.x, and 
fall back when the charset is not a literal Velox accepts.
   
   **Workaround**: `--conf 
spark.gluten.sql.columnar.backend.velox.spark.legacy_coding_error_action=true` 
fixes the unmappable-character case. Aliases still fail.
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   See above.
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to