arsathS opened a new pull request, #24266:
URL: https://github.com/apache/datafusion/pull/24266
## Which issue does this PR close?
- Part of #15914
## Rationale for this change
Spark's `to_binary(str[, fmt])` converts a string to binary, choosing how
based
on `fmt`: `hex`, `utf-8`/`utf8`, or `base64`. `try_to_binary` is the same
function except that input it cannot convert yields NULL instead of raising
an
error.
Neither is available in the `datafusion-spark` crate, and no core DataFusion
function can stand in for them:
- core `decode` accepts only `base64`, `base64pad` and `hex`, so there is no
way to ask it for Spark's `utf-8` conversion
- core `decode` always raises on input it cannot convert, so there is nothing
to build `try_to_binary`'s NULL-returning behaviour from
## What changes are included in this PR?
- New `SparkToBinary` and `SparkTryToBinary` UDFs in
`datafusion/spark/src/function/string/to_binary.rs`
- `to_binary(str[, fmt])` where `fmt` is a case-insensitive literal of
`hex`, `utf-8`, `utf8` or `base64`, defaulting to `hex`
- `hex` decodes two characters per byte, left-padding an odd-length input
with `0`, matching `Unhex`
- `utf-8` / `utf8` returns the string's own UTF-8 bytes
- `base64` matches Java's MIME decoder, which Spark uses: the standard
alphabet, padding optional, and the unused trailing bits of a short final
group ignored
- A NULL value or a NULL `fmt` yields NULL in both functions
- `to_binary` raises on a malformed value or an unrecognised `fmt`;
`try_to_binary` returns NULL for both, matching `nullOnInvalidFormat`
- `fmt` must be foldable, as Spark requires
- `unhex_scalar` is made `pub(crate)` and reused for the `hex` path rather
than duplicating the decoder
- Registered in `mod.rs` (`make_udf_function!`, `export_functions!`,
`functions()`)
- Filled in the commented-out SLT scaffolds for both functions
## Are these changes tested?
Yes.
- 10 unit tests in `to_binary.rs` (each format, hex as the default, empty
input, NULL value and NULL format, invalid value, invalid format, column
input, and a column where one row is invalid)
- 15 SLT assertions across `spark/string/to_binary.slt` and
`spark/string/try_to_binary.slt`
## Are there any user-facing changes?
No. These are new functions in the `datafusion-spark` crate only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
