rdtr opened a new pull request, #13572:
URL: https://github.com/apache/gluten/pull/13572

   ## What changes are proposed in this pull request?
   
   Spark 4.0 added `try_url_decode` (SPARK-48865), which returns NULL on an 
invalid `%` escape. Spark replaces it with `StaticInvoke(UrlCodec, "decode", 
Seq(child, Literal(false)))`, and `UrlCodec.decode` returns NULL when 
`failOnError` is false. Gluten mapped every `UrlCodec.decode` call to Velox 
`url_decode` with only the first argument. Velox's `url_decode` always throws, 
so `try_url_decode` on a column with `http%3A%2F%2spark.apache.org` failed with 
`Illegal hex characters in escape (%) pattern: 2s` instead of returning NULL.
   
   `ExpressionConverter.replaceStaticInvokeWithExpressionTransformer` now 
throws `GlutenNotSupportException` for `UrlCodec.decode` when the second 
argument is `Literal(false)`, so the project falls back to vanilla Spark. 
`url_decode` (`failOnError = true`) and `url_encode` are offloaded as before. 
On Spark 3.5 the second argument of `UrlCodec.decode` is the charset literal 
`"UTF-8"`, so 3.5 is unchanged.
   
   Wrapping `url_decode` in Velox's `try` would turn errors from the child 
expression into NULL as well, which Spark does not do. The fallback can be 
removed once Velox has a sparksql `try_url_decode` (tracked in 
facebookincubator/velox#20057).
   
   Fixes #13571
   
   ## How was this patch tested?
   
   Added `try_url_decode falls back` to `ScalarFunctionsValidateSuite` (Spark 
4.0+). It reads valid, invalid (`%2s`), truncated (`abc%2`) and NULL inputs 
from Parquet, checks that the result equals vanilla Spark and that the project 
runs as `ProjectExec`. Ran the `url_*` tests of the suite on Spark 4.1 with 
`SPARK_ANSI_SQL_MODE=false`. Without the change the new test fails with the 
`VeloxUserError` above. Compiled on Spark 3.5 (Scala 2.12) as well.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to