Stephen Boesch created SPARK-4775:
-------------------------------------
Summary: Possible problem in a simple join? Getting duplicate
rows and missing rows
Key: SPARK-4775
URL: https://issues.apache.org/jira/browse/SPARK-4775
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.3.0
Environment: Run on Mac but should be agnostic
Reporter: Stephen Boesch
I am working on testing of HBase joins. As part of this work some simple
vanilla SparkSQL tests were created. Some of the results are surprising: here
are the details:
------------------------------------
Consider the following schema that includes two columns:
case class JoinTable2Cols(intcol: Int, strcol: String)
Let us register two temp tables using this schema and insert 2 rows and 4 rows
respectively:
val rdd1 = sc.parallelize((1 to 2).map { ix => JoinTable2Cols(ix,
s"valA$ix")})
rdd1.registerTempTable("SparkJoinTable1")
val ids = Seq((1, 1), (1, 2), (2, 3), (2, 4))
val rdd2 = sc.parallelize(ids.map { case (ix, is) => JoinTable2Cols(ix,
s"valB$is")})
val table2 = rdd2.registerTempTable("SparkJoinTable2")
Here is the data in both tables:
Table1 Contents:
[1,valA1]
[2,valA2]
Table2 Contents:
[1,valB1]
[1,valB2]
[2,valB3]
[2,valB4]
Now let us join the tables on the first column:
select t1.intcol t1intcol, t2.intcol t2intcol, t1.strcol t1strcol,
t2.strcol t2strcol from SparkJoinTable1 t1 JOIN
SparkJoinTable2 t2 on t1.intcol = t2.intcol
What results do we get:
came back with 4 results
Results
[1,1,valA1,valB2]
[1,1,valA1,valB2]
[2,2,valA2,valB4]
[2,2,valA2,valB4]
Huh??
Where did valB1 and valB3 go? Why do we have duplicate rows?
Note: the expected results were:
Seq(1, 1, "valA1", "valB1"),
Seq(1, 1, "valA1", "valB2"),
Seq(2, 2, "valA2", "valB3"),
Seq(2, 2, "valA2", "valB4"))
A standalone testing program is attached SparkSQLJoinSuite. An abridged
version of the actual output is also attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
