Yves Li created SPARK-38603:
-------------------------------
Summary: Qualified star selection produces duplicated common
columns after join then alias
Key: SPARK-38603
URL: https://issues.apache.org/jira/browse/SPARK-38603
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.0
Environment: OS: Ubuntu 18.04.5 LTS
Scala version: 2.12.15
Reporter: Yves Li
When joining two DataFrames and then aliasing the result, selecting columns
from the resulting Dataset by a qualified star produces duplicates of the
joined columns.
{code:scala}
scala> val df1 = Seq((1, 10), (2, 20)).toDF("a", "x")
df1: org.apache.spark.sql.DataFrame = [a: int, x: int]
scala> val df2 = Seq((2, 200), (3, 300)).toDF("a", "y")
df2: org.apache.spark.sql.DataFrame = [a: int, y: int]
scala> val joined = df1.join(df2, "a").alias("joined")
joined: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [a: int, x:
int ... 1 more field]
scala> joined.select("*").show()
+---+---+---+
| a| x| y|
+---+---+---+
| 2| 20|200|
+---+---+---+
scala> joined.select("joined.*").show()
+---+---+---+---+
| a| a| x| y|
+---+---+---+---+
| 2| 2| 20|200|
+---+---+---+---+
scala> joined.select("*").select("joined.*").show()
+---+---+---+
| a| x| y|
+---+---+---+
| 2| 20|200|
+---+---+---+ {code}
This appears to be introduced by SPARK-34527, leading to some surprising
behaviour. Using an earlier version, such as Spark 3.0.2, produces the same
output for all three {{{}show(){}}}s.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
