[ https://issues.apache.org/jira/browse/SPARK-40308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruce Robbins updated SPARK-40308: ---------------------------------- Summary: str_to_map should accept non-foldable delimiter arguments (was: str_to_map should accept non-foldable delimiter parameters) > str_to_map should accept non-foldable delimiter arguments > --------------------------------------------------------- > > Key: SPARK-40308 > URL: https://issues.apache.org/jira/browse/SPARK-40308 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: Bruce Robbins > Priority: Minor > > Currently, str_to_map requires the delimiter parameters to be foldable > expressions. For example, the following doesn't work in Spark SQL: > {noformat} > drop table if exists maptbl; > create table maptbl as select ',' as del1, ':' as del2, 'a:1,b:2,c:3' as str; > insert into table maptbl select '%' as del1, '-' as del2, 'a-1%b-2%c-3' as > str; > select str, str_to_map(str, del1, del2) from maptbl; > {noformat} > You get the following error: > {noformat} > str_to_map's delimiters must be foldable.; line 1 pos 12; > {noformat} > However, the above example SQL statements do work in Hive 2.3.9. There, you > get: > {noformat} > +--------------+----------------------------+ > | str | _c1 | > +--------------+----------------------------+ > | a:1,b:2,c:3 | {"a":"1","b":"2","c":"3"} | > | a-1%b-2%c-3 | {"a":"1","b":"2","c":"3"} | > +--------------+----------------------------+ > 2 rows selected (0.13 seconds) > {noformat} > It's unlikely that an input table would have the needed delimiters in > columns. The use-case is more likely to be something like this, where the > delimiters are determined based on some other value: > {noformat} > select > str, > str_to_map(str, ',', if(region = 0, ':', '#')) as m > from > maptbl2; > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org