rdtr opened a new issue, #13641:
URL: https://github.com/apache/gluten/issues/13641
### Backend
VL (Velox)
### Bug description
### Bug description
**[Expected behavior]** On Spark 3.4 and 3.5, `Encode` is a plain expression
that calls `String.getBytes(charset)` (v3.5.9
`stringExpressions.scala:2706-2709`). Characters the charset cannot represent
become `?`, and any charset name or alias the JDK knows is accepted.
`encode('日本語', 'US-ASCII')` returns `3F3F3F`, and `encode('café', 'utf8')`
returns `636166C3A9`. Spark 3.x has no config to change either behavior
(`spark.sql.legacy.javaCharsets` and `spark.sql.legacy.codingErrorAction`
arrived in Spark 4.0).
**[Actual behavior]** Since the Velox pin moved to dft-2026_11_05 (#13540),
which brought Velox `encode` (facebookincubator/velox#18534), Gluten offloads
`Encode` through `Sig[Encode]` (`ExpressionMappings.scala:102`). Before that
pin Velox had no `encode` and the expression fell back. Velox `encode` throws
on unmappable characters unless the query config
`spark.legacy_coding_error_action` is set, and Gluten never sets it. The query
fails with:
```
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: encode: input contains a character that cannot be encoded using
'US-ASCII'
```
Velox also accepts only seven charset names (`US-ASCII`, `ISO-8859-1`,
`UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16`, `UTF-32`, case-insensitive), so
`encode(s, 'utf8')` or `encode(s, 'latin1')` fails with `encode: unsupported
charset 'utf8'`.
The Velox docs for `encode` say Spark 3.x must set
`spark.legacy_coding_error_action=true`
(`velox/docs/functions/spark/string.rst`). Spark 4.x is not affected: there
`Encode` is a `RuntimeReplaceable` whose replacement is a `StaticInvoke` that
Gluten does not offload.
Repro on Spark 3.5 with the Velox backend:
```sql
CREATE TABLE t USING parquet AS SELECT * FROM VALUES ('日本語'), ('café') AS
v(s);
SELECT encode(s, 'US-ASCII') FROM t; -- fails
SELECT encode(s, 'utf8') FROM t; -- fails
```
**Request**: set `spark.legacy_coding_error_action=true` on Spark 3.x, and
fall back when the charset is not a literal Velox accepts.
**Workaround**: `--conf
spark.gluten.sql.columnar.backend.velox.spark.legacy_coding_error_action=true`
fixes the unmappable-character case. Aliases still fail.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
See above.
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]