Martin Junghanns created SPARK-23855:
----------------------------------------
Summary: Performing a Join after a CrossJoin can lead to data
corruption
Key: SPARK-23855
URL: https://issues.apache.org/jira/browse/SPARK-23855
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.1, 2.2.0
Reporter: Martin Junghanns
The following tests produces the wrong result for the join operation. The error
only occurs when joining on the first column of the crossed dataframe. However,
a subsequent select fixes the data (which is of course not a solution).
It works on 2.3.0 though. It would be nice to get this fixed on the 2.2.x
releases, too. Maybe someone can point me to the issue that has been fixed?
Would be nice to see the solution in code.
{code}
it("should correctly perform a join after a cross") {
val df1 = sparkSession.createDataFrame(Seq(Tuple1(0L)))
.toDF("a")
val df2 = sparkSession.createDataFrame(Seq(Tuple1(1L)))
.toDF("b")
val df3 = sparkSession.createDataFrame(Seq(Tuple1(0L)))
.toDF("c")
val cross = df1.crossJoin(df2)
cross.show()
val joined = cross
.join(df3, cross.col("a") === df3.col("c"))
joined.show()
val selected = joined.select("*")
selected.show
}
{code}
prints:
{code:java}
+---+---+
| a| b|
+---+---+
| 0| 1|
+---+---+
+---+---+---+
| a| b| c|
+---+---+---+
| 0| 0| 1|
+---+---+---+
+---+---+---+
| a| b| c|
+---+---+---+
| 0| 1| 0|
+---+---+---+
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
