[spark] branch branch-2.4 updated: [SPARK-32693][SQL][2.4] Compare two dataframes with same schema except nullable property
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ef42225 [SPARK-32693][SQL][2.4] Compare two dataframes with same
schema except nullable property
ef42225 is described below
commit ef422257fe47ec0e8a0e94c6671c5d6316944620
Author: Liang-Chi Hsieh
AuthorDate: Sun Aug 30 06:52:15 2020 +0900
[SPARK-32693][SQL][2.4] Compare two dataframes with same schema except
nullable property
### What changes were proposed in this pull request?
This PR changes key data types check in `HashJoin` to use `sameType`. This
backports #29555 to branch-2.4.
### Why are the changes needed?
Looks at the resolving condition of `SetOperation`, it requires only each
left data types should be `sameType` as the right ones. Logically the `EqualTo`
expression in equi-join, also requires only left data type `sameType` as right
data type. Then `HashJoin` requires left keys data type exactly the same as
right keys data type, looks not reasonable.
It makes inconsistent results when doing `except` between two dataframes.
If two dataframes don't have nested fields, even their field nullable
property different, `HashJoin` passes the key type check because it checks
field individually so field nullable property is ignored.
If two dataframes have nested fields like struct, `HashJoin` fails the key
type check because now it compare two struct types and nullable property now
affects.
### Does this PR introduce _any_ user-facing change?
Yes. Making consistent `except` operation between dataframes.
### How was this patch tested?
Unit test.
Closes #29576 from viirya/SPARK-32693-2.4.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Takeshi Yamamuro
---
.../spark/sql/execution/joins/HashJoin.scala | 7 ++--
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 39 ++
2 files changed, 44 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
index b197bf6..141f388a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
@@ -62,8 +62,11 @@ trait HashJoin {
}
protected lazy val (buildKeys, streamedKeys) = {
-require(leftKeys.map(_.dataType) == rightKeys.map(_.dataType),
- "Join keys from two sides should have same types")
+require(leftKeys.length == rightKeys.length &&
+ leftKeys.map(_.dataType)
+.zip(rightKeys.map(_.dataType))
+.forall(types => types._1.sameType(types._2)),
+ "Join keys from two sides should have same length and types")
val lkeys =
HashJoin.rewriteKeyExpr(leftKeys).map(BindReferences.bindReference(_,
left.output))
val rkeys = HashJoin.rewriteKeyExpr(rightKeys)
.map(BindReferences.bindReference(_, right.output))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e6b30f9..9f77314 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -17,12 +17,15 @@
package org.apache.spark.sql
+import scala.collection.JavaConverters._
+
import org.apache.spark.sql.catalyst.plans.{Inner, LeftOuter, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.Join
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
import testImplicits._
@@ -295,4 +298,40 @@ class DataFrameJoinSuite extends QueryTest with
SharedSQLContext {
df.join(df, df("id") <=> df("id")).queryExecution.optimizedPlan
}
}
+
+ test("SPARK-32693: Compare two dataframes with same schema except nullable
property") {
+val schema1 = StructType(
+ StructField("a", IntegerType, false) ::
+StructField("b", IntegerType, false) ::
+StructField("c", IntegerType, false) :: Nil)
+val rowSeq1: List[Row] = List(Row(10, 1, 1), Row(10, 50, 2))
+val df1 = spark.createDataFrame(rowSeq1.asJava, schema1)
+
+val schema2 = StructType(
+ StructField("a", IntegerType) ::
+StructField("b", IntegerType) ::
+StructField("c", IntegerType) :: Nil)
+val rowSeq2: List[Row] = List(Row(10, 1, 1))
+val df2 = spark.createDataFrame(rowSeq2.asJava, schema2)
+
+checkAnswer(d
[spark] branch branch-2.4 updated: [SPARK-32693][SQL][2.4] Compare two dataframes with same schema except nullable property
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ef42225 [SPARK-32693][SQL][2.4] Compare two dataframes with same
schema except nullable property
ef42225 is described below
commit ef422257fe47ec0e8a0e94c6671c5d6316944620
Author: Liang-Chi Hsieh
AuthorDate: Sun Aug 30 06:52:15 2020 +0900
[SPARK-32693][SQL][2.4] Compare two dataframes with same schema except
nullable property
### What changes were proposed in this pull request?
This PR changes key data types check in `HashJoin` to use `sameType`. This
backports #29555 to branch-2.4.
### Why are the changes needed?
Looks at the resolving condition of `SetOperation`, it requires only each
left data types should be `sameType` as the right ones. Logically the `EqualTo`
expression in equi-join, also requires only left data type `sameType` as right
data type. Then `HashJoin` requires left keys data type exactly the same as
right keys data type, looks not reasonable.
It makes inconsistent results when doing `except` between two dataframes.
If two dataframes don't have nested fields, even their field nullable
property different, `HashJoin` passes the key type check because it checks
field individually so field nullable property is ignored.
If two dataframes have nested fields like struct, `HashJoin` fails the key
type check because now it compare two struct types and nullable property now
affects.
### Does this PR introduce _any_ user-facing change?
Yes. Making consistent `except` operation between dataframes.
### How was this patch tested?
Unit test.
Closes #29576 from viirya/SPARK-32693-2.4.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Takeshi Yamamuro
---
.../spark/sql/execution/joins/HashJoin.scala | 7 ++--
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 39 ++
2 files changed, 44 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
index b197bf6..141f388a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
@@ -62,8 +62,11 @@ trait HashJoin {
}
protected lazy val (buildKeys, streamedKeys) = {
-require(leftKeys.map(_.dataType) == rightKeys.map(_.dataType),
- "Join keys from two sides should have same types")
+require(leftKeys.length == rightKeys.length &&
+ leftKeys.map(_.dataType)
+.zip(rightKeys.map(_.dataType))
+.forall(types => types._1.sameType(types._2)),
+ "Join keys from two sides should have same length and types")
val lkeys =
HashJoin.rewriteKeyExpr(leftKeys).map(BindReferences.bindReference(_,
left.output))
val rkeys = HashJoin.rewriteKeyExpr(rightKeys)
.map(BindReferences.bindReference(_, right.output))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e6b30f9..9f77314 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -17,12 +17,15 @@
package org.apache.spark.sql
+import scala.collection.JavaConverters._
+
import org.apache.spark.sql.catalyst.plans.{Inner, LeftOuter, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.Join
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
import testImplicits._
@@ -295,4 +298,40 @@ class DataFrameJoinSuite extends QueryTest with
SharedSQLContext {
df.join(df, df("id") <=> df("id")).queryExecution.optimizedPlan
}
}
+
+ test("SPARK-32693: Compare two dataframes with same schema except nullable
property") {
+val schema1 = StructType(
+ StructField("a", IntegerType, false) ::
+StructField("b", IntegerType, false) ::
+StructField("c", IntegerType, false) :: Nil)
+val rowSeq1: List[Row] = List(Row(10, 1, 1), Row(10, 50, 2))
+val df1 = spark.createDataFrame(rowSeq1.asJava, schema1)
+
+val schema2 = StructType(
+ StructField("a", IntegerType) ::
+StructField("b", IntegerType) ::
+StructField("c", IntegerType) :: Nil)
+val rowSeq2: List[Row] = List(Row(10, 1, 1))
+val df2 = spark.createDataFrame(rowSeq2.asJava, schema2)
+
+checkAnswer(d
[spark] branch branch-2.4 updated: [SPARK-32693][SQL][2.4] Compare two dataframes with same schema except nullable property
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ef42225 [SPARK-32693][SQL][2.4] Compare two dataframes with same
schema except nullable property
ef42225 is described below
commit ef422257fe47ec0e8a0e94c6671c5d6316944620
Author: Liang-Chi Hsieh
AuthorDate: Sun Aug 30 06:52:15 2020 +0900
[SPARK-32693][SQL][2.4] Compare two dataframes with same schema except
nullable property
### What changes were proposed in this pull request?
This PR changes key data types check in `HashJoin` to use `sameType`. This
backports #29555 to branch-2.4.
### Why are the changes needed?
Looks at the resolving condition of `SetOperation`, it requires only each
left data types should be `sameType` as the right ones. Logically the `EqualTo`
expression in equi-join, also requires only left data type `sameType` as right
data type. Then `HashJoin` requires left keys data type exactly the same as
right keys data type, looks not reasonable.
It makes inconsistent results when doing `except` between two dataframes.
If two dataframes don't have nested fields, even their field nullable
property different, `HashJoin` passes the key type check because it checks
field individually so field nullable property is ignored.
If two dataframes have nested fields like struct, `HashJoin` fails the key
type check because now it compare two struct types and nullable property now
affects.
### Does this PR introduce _any_ user-facing change?
Yes. Making consistent `except` operation between dataframes.
### How was this patch tested?
Unit test.
Closes #29576 from viirya/SPARK-32693-2.4.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Takeshi Yamamuro
---
.../spark/sql/execution/joins/HashJoin.scala | 7 ++--
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 39 ++
2 files changed, 44 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
index b197bf6..141f388a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
@@ -62,8 +62,11 @@ trait HashJoin {
}
protected lazy val (buildKeys, streamedKeys) = {
-require(leftKeys.map(_.dataType) == rightKeys.map(_.dataType),
- "Join keys from two sides should have same types")
+require(leftKeys.length == rightKeys.length &&
+ leftKeys.map(_.dataType)
+.zip(rightKeys.map(_.dataType))
+.forall(types => types._1.sameType(types._2)),
+ "Join keys from two sides should have same length and types")
val lkeys =
HashJoin.rewriteKeyExpr(leftKeys).map(BindReferences.bindReference(_,
left.output))
val rkeys = HashJoin.rewriteKeyExpr(rightKeys)
.map(BindReferences.bindReference(_, right.output))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e6b30f9..9f77314 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -17,12 +17,15 @@
package org.apache.spark.sql
+import scala.collection.JavaConverters._
+
import org.apache.spark.sql.catalyst.plans.{Inner, LeftOuter, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.Join
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
import testImplicits._
@@ -295,4 +298,40 @@ class DataFrameJoinSuite extends QueryTest with
SharedSQLContext {
df.join(df, df("id") <=> df("id")).queryExecution.optimizedPlan
}
}
+
+ test("SPARK-32693: Compare two dataframes with same schema except nullable
property") {
+val schema1 = StructType(
+ StructField("a", IntegerType, false) ::
+StructField("b", IntegerType, false) ::
+StructField("c", IntegerType, false) :: Nil)
+val rowSeq1: List[Row] = List(Row(10, 1, 1), Row(10, 50, 2))
+val df1 = spark.createDataFrame(rowSeq1.asJava, schema1)
+
+val schema2 = StructType(
+ StructField("a", IntegerType) ::
+StructField("b", IntegerType) ::
+StructField("c", IntegerType) :: Nil)
+val rowSeq2: List[Row] = List(Row(10, 1, 1))
+val df2 = spark.createDataFrame(rowSeq2.asJava, schema2)
+
+checkAnswer(d
[spark] branch branch-2.4 updated: [SPARK-32693][SQL][2.4] Compare two dataframes with same schema except nullable property
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ef42225 [SPARK-32693][SQL][2.4] Compare two dataframes with same
schema except nullable property
ef42225 is described below
commit ef422257fe47ec0e8a0e94c6671c5d6316944620
Author: Liang-Chi Hsieh
AuthorDate: Sun Aug 30 06:52:15 2020 +0900
[SPARK-32693][SQL][2.4] Compare two dataframes with same schema except
nullable property
### What changes were proposed in this pull request?
This PR changes key data types check in `HashJoin` to use `sameType`. This
backports #29555 to branch-2.4.
### Why are the changes needed?
Looks at the resolving condition of `SetOperation`, it requires only each
left data types should be `sameType` as the right ones. Logically the `EqualTo`
expression in equi-join, also requires only left data type `sameType` as right
data type. Then `HashJoin` requires left keys data type exactly the same as
right keys data type, looks not reasonable.
It makes inconsistent results when doing `except` between two dataframes.
If two dataframes don't have nested fields, even their field nullable
property different, `HashJoin` passes the key type check because it checks
field individually so field nullable property is ignored.
If two dataframes have nested fields like struct, `HashJoin` fails the key
type check because now it compare two struct types and nullable property now
affects.
### Does this PR introduce _any_ user-facing change?
Yes. Making consistent `except` operation between dataframes.
### How was this patch tested?
Unit test.
Closes #29576 from viirya/SPARK-32693-2.4.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Takeshi Yamamuro
---
.../spark/sql/execution/joins/HashJoin.scala | 7 ++--
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 39 ++
2 files changed, 44 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
index b197bf6..141f388a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
@@ -62,8 +62,11 @@ trait HashJoin {
}
protected lazy val (buildKeys, streamedKeys) = {
-require(leftKeys.map(_.dataType) == rightKeys.map(_.dataType),
- "Join keys from two sides should have same types")
+require(leftKeys.length == rightKeys.length &&
+ leftKeys.map(_.dataType)
+.zip(rightKeys.map(_.dataType))
+.forall(types => types._1.sameType(types._2)),
+ "Join keys from two sides should have same length and types")
val lkeys =
HashJoin.rewriteKeyExpr(leftKeys).map(BindReferences.bindReference(_,
left.output))
val rkeys = HashJoin.rewriteKeyExpr(rightKeys)
.map(BindReferences.bindReference(_, right.output))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e6b30f9..9f77314 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -17,12 +17,15 @@
package org.apache.spark.sql
+import scala.collection.JavaConverters._
+
import org.apache.spark.sql.catalyst.plans.{Inner, LeftOuter, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.Join
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
import testImplicits._
@@ -295,4 +298,40 @@ class DataFrameJoinSuite extends QueryTest with
SharedSQLContext {
df.join(df, df("id") <=> df("id")).queryExecution.optimizedPlan
}
}
+
+ test("SPARK-32693: Compare two dataframes with same schema except nullable
property") {
+val schema1 = StructType(
+ StructField("a", IntegerType, false) ::
+StructField("b", IntegerType, false) ::
+StructField("c", IntegerType, false) :: Nil)
+val rowSeq1: List[Row] = List(Row(10, 1, 1), Row(10, 50, 2))
+val df1 = spark.createDataFrame(rowSeq1.asJava, schema1)
+
+val schema2 = StructType(
+ StructField("a", IntegerType) ::
+StructField("b", IntegerType) ::
+StructField("c", IntegerType) :: Nil)
+val rowSeq2: List[Row] = List(Row(10, 1, 1))
+val df2 = spark.createDataFrame(rowSeq2.asJava, schema2)
+
+checkAnswer(d
[spark] branch branch-2.4 updated: [SPARK-32693][SQL][2.4] Compare two dataframes with same schema except nullable property
This is an automated email from the ASF dual-hosted git repository.
yamamuro pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ef42225 [SPARK-32693][SQL][2.4] Compare two dataframes with same
schema except nullable property
ef42225 is described below
commit ef422257fe47ec0e8a0e94c6671c5d6316944620
Author: Liang-Chi Hsieh
AuthorDate: Sun Aug 30 06:52:15 2020 +0900
[SPARK-32693][SQL][2.4] Compare two dataframes with same schema except
nullable property
### What changes were proposed in this pull request?
This PR changes key data types check in `HashJoin` to use `sameType`. This
backports #29555 to branch-2.4.
### Why are the changes needed?
Looks at the resolving condition of `SetOperation`, it requires only each
left data types should be `sameType` as the right ones. Logically the `EqualTo`
expression in equi-join, also requires only left data type `sameType` as right
data type. Then `HashJoin` requires left keys data type exactly the same as
right keys data type, looks not reasonable.
It makes inconsistent results when doing `except` between two dataframes.
If two dataframes don't have nested fields, even their field nullable
property different, `HashJoin` passes the key type check because it checks
field individually so field nullable property is ignored.
If two dataframes have nested fields like struct, `HashJoin` fails the key
type check because now it compare two struct types and nullable property now
affects.
### Does this PR introduce _any_ user-facing change?
Yes. Making consistent `except` operation between dataframes.
### How was this patch tested?
Unit test.
Closes #29576 from viirya/SPARK-32693-2.4.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Takeshi Yamamuro
---
.../spark/sql/execution/joins/HashJoin.scala | 7 ++--
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 39 ++
2 files changed, 44 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
index b197bf6..141f388a 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala
@@ -62,8 +62,11 @@ trait HashJoin {
}
protected lazy val (buildKeys, streamedKeys) = {
-require(leftKeys.map(_.dataType) == rightKeys.map(_.dataType),
- "Join keys from two sides should have same types")
+require(leftKeys.length == rightKeys.length &&
+ leftKeys.map(_.dataType)
+.zip(rightKeys.map(_.dataType))
+.forall(types => types._1.sameType(types._2)),
+ "Join keys from two sides should have same length and types")
val lkeys =
HashJoin.rewriteKeyExpr(leftKeys).map(BindReferences.bindReference(_,
left.output))
val rkeys = HashJoin.rewriteKeyExpr(rightKeys)
.map(BindReferences.bindReference(_, right.output))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
index e6b30f9..9f77314 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala
@@ -17,12 +17,15 @@
package org.apache.spark.sql
+import scala.collection.JavaConverters._
+
import org.apache.spark.sql.catalyst.plans.{Inner, LeftOuter, RightOuter}
import org.apache.spark.sql.catalyst.plans.logical.Join
import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
import testImplicits._
@@ -295,4 +298,40 @@ class DataFrameJoinSuite extends QueryTest with
SharedSQLContext {
df.join(df, df("id") <=> df("id")).queryExecution.optimizedPlan
}
}
+
+ test("SPARK-32693: Compare two dataframes with same schema except nullable
property") {
+val schema1 = StructType(
+ StructField("a", IntegerType, false) ::
+StructField("b", IntegerType, false) ::
+StructField("c", IntegerType, false) :: Nil)
+val rowSeq1: List[Row] = List(Row(10, 1, 1), Row(10, 50, 2))
+val df1 = spark.createDataFrame(rowSeq1.asJava, schema1)
+
+val schema2 = StructType(
+ StructField("a", IntegerType) ::
+StructField("b", IntegerType) ::
+StructField("c", IntegerType) :: Nil)
+val rowSeq2: List[Row] = List(Row(10, 1, 1))
+val df2 = spark.createDataFrame(rowSeq2.asJava, schema2)
+
+checkAnswer(d
