Bruce Robbins created SPARK-40308:
-------------------------------------
Summary: str_to_map should accept non-foldable delimiter parameters
Key: SPARK-40308
URL: https://issues.apache.org/jira/browse/SPARK-40308
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: Bruce Robbins
Currently, str_to_map requires the delimiter parameters to be foldable
expressions. For example, the following doesn't work in Spark SQL:
{noformat}
drop table if exists maptbl;
create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str;
insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as str;
select str, str_to_map(str, del1, del2) from maptbl;
{noformat}
You get the following error:
{noformat}
str_to_map's delimiters must be foldable.; line 1 pos 12;
{noformat}
However, the above example SQL statements do work in Hive 2.3.9. There, you get:
{noformat}
+--------------+----------------------------+
| str | _c1 |
+--------------+----------------------------+
| a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} |
| a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} |
+--------------+----------------------------+
2 rows selected (0.13 seconds)
{noformat}
It's unlikely that an input table would have the needed delimiters in columns.
The use-case is more likely to be something like this, where the delimiters are
determined based on some other value:
{noformat}
select
str,
str_to_map(str, ',', if(region = 0, ':', '#')) as m
from
maptbl2;
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
