david-mollitor-db opened a new pull request, #58679: URL: https://github.com/apache/spark/pull/58679
### What changes were proposed in this pull request? `UnBase64` (SQL `unbase64`, and `try_to_binary(..., 'base64')`) decodes a base64 string to bytes. Both its interpreted path and its generated code called `.toString()` on the input `UTF8String` before decoding, which decodes the whole input from UTF-8 into a Java `String` on every row purely to hand it to the decoder. `java.util.Base64.Decoder` accepts a `byte[]` directly, so this PR passes the raw UTF-8 bytes (`UTF8String.getBytes()`) in both paths and drops the `.toString()`: - interpreted (`nullSafeEval`): `...getMimeDecoder.decode(string.asInstanceOf[UTF8String].getBytes)` - codegen (`doGenCode`): `...getMimeDecoder().decode($child.getBytes())` `UnBase64.isValidBase64` (used only by the `try_to_binary` validation path) still iterates the string form and is left unchanged; converting it to iterate bytes is a separate, larger change and out of scope here. ### Why are the changes needed? It removes a per-row `String` allocation and UTF-8 decode on a hot path. The bytes are already available on the `UTF8String`, and the decoder consumes bytes, so materializing an intermediate Java `String` just to feed the decoder is unnecessary. The change is behavior-preserving. `getMimeDecoder().decode(String)` internally converts the string to bytes with ISO-8859-1 before decoding. For valid base64 -- which is always ASCII -- those bytes are identical to the raw UTF-8 bytes. For any non-ASCII or otherwise non-base64 content, the MIME decoder ignores every byte outside the base64 alphabet (`A-Za-z0-9+/=`) under both paths, so the decoded output is the same. The result is therefore identical for all inputs. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `StringExpressionsSuite` base64/unbase64 tests pass (round-trip, empty, null, `failOnError`, and chunked/non-chunked encodings). Added an assertion whose base64 input embeds a multi-byte character, confirming the MIME decoder still ignores it and the result is unchanged. That same assertion was also run against the original `toString()`-based code and passes, confirming the behavior is unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
