[
https://issues.apache.org/jira/browse/SPARK-55560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059215#comment-18059215
]
Uroš Bojanić commented on SPARK-55560:
--------------------------------------
+1 with [~yiconghuang], `pairDelim` and `keyValueDelim` are treated as regular
expressions.
[~shaobog] Please see:
[https://docs.databricks.com/gcp/en/sql/language-manual/functions/str_to_map]
In case you want to treat it as literal string, then just escape the pipe
character like this: '\\|'
Happy to discuss more if needed, but otherwise I recommend resolving this issue
and closing the bug report. Thanks everyone!
> STR_TO_MAP returns wrong result with "|" in delimiter
> -----------------------------------------------------
>
> Key: SPARK-55560
> URL: https://issues.apache.org/jira/browse/SPARK-55560
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.6
> Reporter: Shaobo Guan
> Priority: Major
>
> STR_TO_MAP returns wrong result with "|" in delimiter:
>
> ('key1|value1,key2|value2', ',', '|'):
> Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most
> recent failure: Lost task 0.0 in stage 0.0 (TID 0)
> (ip-10-0-3-243.us-east-2.compute.internal executor driver):
> org.apache.spark.SparkRuntimeException: [DUPLICATED_MAP_KEY] Duplicate map
> key k was found, please check the input data. If you want to remove the
> duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so
> that the key inserted at last takes precedence.
> at
> org.apache.spark.sql.errors.QueryExecutionErrors$.duplicateMapKeyFoundError(QueryExecutionErrors.scala:1278)
> at
> org.apache.spark.sql.catalyst.util.ArrayBasedMapBuilder.put(ArrayBasedMapBuilder.scala:69)
>
> Row 11 (inputs=(k10|푇ͳõ, &, |)):
> Column 'out': actual=Map(k -> 10|푇ͳõ), expected=Map(k10 -> 푇ͳõ)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
