Vitalii Li created SPARK-40112:
----------------------------------
Summary: Improve the TO_BINARY() function
Key: SPARK-40112
URL: https://issues.apache.org/jira/browse/SPARK-40112
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: Vitalii Li
Original description SPARK-37507:
{quote}to_binary(expr, fmt) is a common function available in many other
systems to provide a unified entry for string to binary data conversion, where
fmt can be utf8, base64, hex and base2 (or whatever the reverse operation
to_char()supports).
[https://docs.aws.amazon.com/redshift/latest/dg/r_TO_VARBYTE.html]
[https://docs.snowflake.com/en/sql-reference/functions/to_binary.html]
[https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#format_string_as_bytes]
[https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/etRo5aTAY9n5fUPjxSEynw]
Related Spark functions: unbase64, unhex
{quote}
Expected improvement:
* `base64` behaves more strictly, i.e. does not allow symbols not included in
base64 dictionary (A-Za-z0-9+/) and verifies correct padding and symbol groups
(see RFC 4648 § 4). Whitespaces are ignored.
** Current implementation allows arbitrary strings and invalid symbols are
skipped.
* `hex` converts only valid hexadecimal strings and throws errors otherwise.
Whitespaces are not allowed.
* `utf-8` and `utf8` are interchangeable.
* Correct errors are thrown and classified for invalid input and invalid
format:
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
