Terry Kim created SPARK-29664:
---------------------------------
Summary: Column.getItem behavior is not consistent with Scala
version
Key: SPARK-29664
URL: https://issues.apache.org/jira/browse/SPARK-29664
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.0.0
Reporter: Terry Kim
In PySpark, Column.getItem's behavior is different from the Scala version.
For example,
In PySpark:
{code:python}
df = spark.range(2)
map_col = create_map(lit(0), lit(100), lit(1), lit(200))
df.withColumn("mapped", map_col.getItem(col('id'))).show()
# +---+------+
# | id|mapped|
# +---+------+
# | 0| 100|
# | 1| 200|
# +---+------+
{code}
In Scala:
{code:scala}
val df = spark.range(2)
val map_col = map(lit(0), lit(100), lit(1), lit(200))
// The following getItem results in the following exception, which is the right
behavior:
// java.lang.RuntimeException: Unsupported literal type class
org.apache.spark.sql.Column id
// at
org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:78)
// at org.apache.spark.sql.Column.getItem(Column.scala:856)
// ... 49 elided
df.withColumn("mapped", map_col.getItem(col("id"))).show
// You have to use apply() to match with PySpark's behavior.
df.withColumn("mapped", map_col(col("id"))).show
// +---+------+
// | id|mapped|
// +---+------+
// | 0| 100|
// | 1| 200|
// +---+------+
{code}
Looking at the code for Scala implementation, PySpark's behavior is incorrect
since the argument to getItem becomes `Literal`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
