[
https://issues.apache.org/jira/browse/SPARK-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-20912:
------------------------------------
Assignee: (was: Apache Spark)
> map function with columns as strings
> ------------------------------------
>
> Key: SPARK-20912
> URL: https://issues.apache.org/jira/browse/SPARK-20912
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
> Priority: Trivial
>
> There's only {{map}} function that accepts {{Column}} values only. It'd be
> very helpful to have a variant that accepted {{String}} for columns like
> {{array}} or {{struct}}.
> {code}
> scala> val kvs = Seq(("key", "value")).toDF("k", "v")
> kvs: org.apache.spark.sql.DataFrame = [k: string, v: string]
> scala> kvs.printSchema
> root
> |-- k: string (nullable = true)
> |-- v: string (nullable = true)
> scala> kvs.withColumn("map", map("k", "v")).show
> <console>:26: error: type mismatch;
> found : String("k")
> required: org.apache.spark.sql.Column
> kvs.withColumn("map", map("k", "v")).show
> ^
> <console>:26: error: type mismatch;
> found : String("v")
> required: org.apache.spark.sql.Column
> kvs.withColumn("map", map("k", "v")).show
> ^
> // note $ to create Columns per string
> // not very dev-friendly
> scala> kvs.withColumn("map", map($"k", $"v")).show
> +---+-----+-----------------+
> | k| v| map|
> +---+-----+-----------------+
> |key|value|Map(key -> value)|
> +---+-----+-----------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
