[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r143598538
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -412,7 +480,66 @@ object TypeCoercion {
val commonTypes = lhs.zip(rhs).flatMap { case (l, r) =>
findCommonTypeForBinaryComparison(l.dataType, r.dataType)
+ .orElse(findTightestCommonType(l.dataType, r.dataType))
--- End diff --
Could you still try to avoid duplicating the codes?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r140802924 --- Diff: docs/sql-programming-guide.md --- @@ -1460,6 +1460,13 @@ that these options will be deprecated in future release as more optimizations ar Configures the number of partitions to use when shuffling data for joins or aggregations. + +spark.sql.typeCoercion.mode +default + +Whether compatible with Hive. Available options are default and hive. --- End diff -- This description feels inadequate to me. I think most users will think "hive" means "old, legacy way of doing things and "default" means "new, better, spark way of doing things". But I haven't heard an argument in favor of the "default" behavior, just that we don't want to have a breaking change of behavior. So (a) I'd advocate that we rename "default" to "legacy", or something else along those lines. I do think it should be the default value, to avoid changing behavior. and (b) I think the doc section here should more clearly indicate the difference, eg. "The 'legacy' typeCoercion mode was used in spark prior to 2.3, and so it continues to be the default to avoid breaking behavior. However, it has logical inconsistencies. The 'hive' mode is preferred for most new applications, though it may require additional manual casting. I am even wondering if we should have a 3rd option, to not implicit cast across type categories, eg. like postgres, as this avoids nasty surprises for the user. While the casts are convenient, when it doesn't work there is very little indication to the user that anything went wrong -- most likely they'll just keep continue processing data though the results don't actually have the semantics they want. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r140799432
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -115,21 +115,46 @@ object TypeCoercion {
* is a String and the other is not. It also handles when one op is a
Date and the
* other is a Timestamp by making the target type to be String.
*/
- val findCommonTypeForBinaryComparison: (DataType, DataType) =>
Option[DataType] = {
-// We should cast all relative timestamp/date/string comparison into
string comparisons
-// This behaves as a user would expect because timestamp strings sort
lexicographically.
-// i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true
-case (StringType, DateType) => Some(StringType)
-case (DateType, StringType) => Some(StringType)
-case (StringType, TimestampType) => Some(StringType)
-case (TimestampType, StringType) => Some(StringType)
-case (TimestampType, DateType) => Some(StringType)
-case (DateType, TimestampType) => Some(StringType)
-case (StringType, NullType) => Some(StringType)
-case (NullType, StringType) => Some(StringType)
-case (l: StringType, r: AtomicType) if r != StringType => Some(r)
-case (l: AtomicType, r: StringType) if (l != StringType) => Some(l)
-case (l, r) => None
+ private def findCommonTypeForBinaryComparison(
+ plan: LogicalPlan,
+ l: DataType,
+ r: DataType): Option[DataType] =
+if (!plan.conf.isHiveTypeCoercionMode) {
+ (l, r) match {
+// We should cast all relative timestamp/date/string comparison
into string comparisons
+// This behaves as a user would expect because timestamp strings
sort lexicographically.
+// i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true
--- End diff --
I think this comment should be updated given the latest investigation,
explaining both the original motivation, and how it is flawed. Eg.
"Originally spark cast all relative timestamp/date/string comparison into
string comparisons. The motivation was that this would lead to natural
comparisons on simple string inputs for times, eg. TimeStamp(2013-01-01 00:00
...) < "2014" = true. However, this leads to other logical inconsistencies,
eg. TimeStamp(2013-01-01 00:00 ...) < "5" = true. Also, equals is not
consistent with other binary comparisions. Futhermore, comparing to a string
that does not look like a time at all will still compare, just with an
unexpected result."
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r140801036
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2677,4 +2677,142 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
checkAnswer(df, Row(1, 1, 1))
}
}
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
NumericType") {
+withTempView("v") {
+ val str1 = Long.MaxValue.toString + "1"
+ val str2 = Int.MaxValue.toString + "1"
+ val str3 = "10"
+ Seq(str1, str2, str3).toDF("c1").createOrReplaceTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("SELECT c1 from v where c1 > 0"),
+ Row(str1) :: Row(str2) :: Row(str3) :: Nil)
+checkAnswer(sql("SELECT c1 from v where c1 > 0L"),
+ Row(str1) :: Row(str2) :: Row(str3) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("SELECT c1 from v where c1 > 0"), Row(str3) :: Nil)
+checkAnswer(sql("SELECT c1 from v where c1 > 0L"), Row(str2) ::
Row(str3) :: Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: DoubleType vs
IntegerType") {
+withTempView("v") {
+ Seq(("0", 1), ("-0.4", 2), ("0.6", 3)).toDF("c1",
"c2").createOrReplaceTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0L"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0.0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = -0.4"),
Seq(Row("-0.4")))
+checkAnswer(sql("SELECT count(*) FROM v WHERE c1 > 0"), Row(1) ::
Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0"), Seq(Row("0"),
Row("-0.4"), Row("0.6")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0L"), Seq(Row("0"),
Row("-0.4"), Row("0.6")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0.0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = -0.4"),
Seq(Row("-0.4")))
+checkAnswer(sql("SELECT count(*) FROM v WHERE c1 > 0"), Row(0) ::
Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
DateType") {
+withTempView("v") {
+ val v1 = Date.valueOf("2017-09-22")
+ val v2 = Date.valueOf("2017-09-09")
+ Seq(v1, v2).toDF("c1").createTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Row(v1)
:: Row(v2) :: Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
date)"),
+ Row(v1) :: Row(v2) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
date)"),
+ Row(v1) :: Row(v2) :: Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
TimestampType") {
+withTempView("v") {
+ val v1 = Timestamp.valueOf("2017-07-21 23:42:12.123")
+ val v2 = Timestamp.valueOf("2017-08-21 23:42:12.123")
+ Seq(v1, v2).toDF("c1").createTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Row(v2)
:: Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
timestamp)"),
+ Row(v2) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
timestamp)"),
+ Row(v2) :: Nil)
--- End diff --
perhaps there should also be a comparison for a time with more degrees of
precision? It seems from the original discussion in
https://issues.apache.org/jira/browse/SPARK-8420?focusedCommentId=14592654=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14592654
this was one of the concerns, eg. with `'1969-12-31 16:00:00'` `'1969-12-31
16:00:00.0'`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r140797299
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2677,4 +2677,142 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
checkAnswer(df, Row(1, 1, 1))
}
}
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
NumericType") {
+withTempView("v") {
+ val str1 = Long.MaxValue.toString + "1"
+ val str2 = Int.MaxValue.toString + "1"
+ val str3 = "10"
+ Seq(str1, str2, str3).toDF("c1").createOrReplaceTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("SELECT c1 from v where c1 > 0"),
+ Row(str1) :: Row(str2) :: Row(str3) :: Nil)
+checkAnswer(sql("SELECT c1 from v where c1 > 0L"),
+ Row(str1) :: Row(str2) :: Row(str3) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("SELECT c1 from v where c1 > 0"), Row(str3) :: Nil)
+checkAnswer(sql("SELECT c1 from v where c1 > 0L"), Row(str2) ::
Row(str3) :: Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: DoubleType vs
IntegerType") {
+withTempView("v") {
+ Seq(("0", 1), ("-0.4", 2), ("0.6", 3)).toDF("c1",
"c2").createOrReplaceTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0L"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0.0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = -0.4"),
Seq(Row("-0.4")))
+checkAnswer(sql("SELECT count(*) FROM v WHERE c1 > 0"), Row(1) ::
Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0"), Seq(Row("0"),
Row("-0.4"), Row("0.6")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0L"), Seq(Row("0"),
Row("-0.4"), Row("0.6")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = 0.0"), Seq(Row("0")))
+checkAnswer(sql("SELECT c1 FROM v WHERE c1 = -0.4"),
Seq(Row("-0.4")))
+checkAnswer(sql("SELECT count(*) FROM v WHERE c1 > 0"), Row(0) ::
Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
DateType") {
+withTempView("v") {
+ val v1 = Date.valueOf("2017-09-22")
+ val v2 = Date.valueOf("2017-09-09")
+ Seq(v1, v2).toDF("c1").createTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Row(v1)
:: Row(v2) :: Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
date)"),
+ Row(v1) :: Row(v2) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
date)"),
+ Row(v1) :: Row(v2) :: Nil)
+ }
+}
+ }
+
+ test("SPARK-21646: CommonTypeForBinaryComparison: StringType vs
TimestampType") {
+withTempView("v") {
+ val v1 = Timestamp.valueOf("2017-07-21 23:42:12.123")
+ val v2 = Timestamp.valueOf("2017-08-21 23:42:12.123")
+ Seq(v1, v2).toDF("c1").createTempView("v")
+ withSQLConf(SQLConf.typeCoercionMode.key -> "hive") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Row(v2)
:: Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
timestamp)"),
+ Row(v2) :: Nil)
+ }
+
+ withSQLConf(SQLConf.typeCoercionMode.key -> "default") {
+checkAnswer(sql("select c1 from v where c1 > '2017-8-1'"), Nil)
+checkAnswer(sql("select c1 from v where c1 > cast('2017-8-1' as
timestamp)"),
+ Row(v2) :: Nil)
--- End diff --
I think we should include a comparison which is only the year, eg. ` >
'2014'`, as that was listed as the motivation for the "default" behavior in the
code comments.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r139821887 --- Diff: docs/sql-programming-guide.md --- @@ -968,6 +968,13 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession + + spark.sql.autoTypeCastingCompatibility + false --- End diff -- `hive, default` hive, default --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18853#discussion_r139821539 --- Diff: docs/sql-programming-guide.md --- @@ -968,6 +968,13 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession + + spark.sql.autoTypeCastingCompatibility --- End diff -- `spark.sql.typeCoercion.mode` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139774828
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -352,11 +374,16 @@ object TypeCoercion {
p.makeCopy(Array(Cast(left, TimestampType), right))
case p @ Equality(left @ TimestampType(), right @ StringType()) =>
p.makeCopy(Array(left, Cast(right, TimestampType)))
-
case p @ BinaryComparison(left, right)
-if findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
+if !plan.conf.binaryComparisonCompatibleWithHive &&
+ findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
val commonType = findCommonTypeForBinaryComparison(left.dataType,
right.dataType).get
p.makeCopy(Array(castExpr(left, commonType), castExpr(right,
commonType)))
+ case p @ BinaryComparison(left, right)
+if plan.conf.binaryComparisonCompatibleWithHive &&
--- End diff --
I am fine to have two separate rules, but we can call the shared functions.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139719600
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -352,11 +374,16 @@ object TypeCoercion {
p.makeCopy(Array(Cast(left, TimestampType), right))
case p @ Equality(left @ TimestampType(), right @ StringType()) =>
p.makeCopy(Array(left, Cast(right, TimestampType)))
-
case p @ BinaryComparison(left, right)
-if findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
+if !plan.conf.binaryComparisonCompatibleWithHive &&
+ findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
val commonType = findCommonTypeForBinaryComparison(left.dataType,
right.dataType).get
p.makeCopy(Array(castExpr(left, commonType), castExpr(right,
commonType)))
+ case p @ BinaryComparison(left, right)
+if plan.conf.binaryComparisonCompatibleWithHive &&
--- End diff --
It needs 2 `PromoteStrings` and 2 `InConversion` if separate from current
rule, So I merge the logic to `findCommonTypeForBinaryComparison`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139465565
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -352,11 +374,16 @@ object TypeCoercion {
p.makeCopy(Array(Cast(left, TimestampType), right))
case p @ Equality(left @ TimestampType(), right @ StringType()) =>
p.makeCopy(Array(left, Cast(right, TimestampType)))
-
case p @ BinaryComparison(left, right)
-if findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
+if !plan.conf.binaryComparisonCompatibleWithHive &&
+ findCommonTypeForBinaryComparison(left.dataType,
right.dataType).isDefined =>
val commonType = findCommonTypeForBinaryComparison(left.dataType,
right.dataType).get
p.makeCopy(Array(castExpr(left, commonType), castExpr(right,
commonType)))
+ case p @ BinaryComparison(left, right)
+if plan.conf.binaryComparisonCompatibleWithHive &&
--- End diff --
This is hard to maintain and debug. Instead of mixing them together, could
you separate it from the current rule Spark uses?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139464749
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -925,6 +925,12 @@ object SQLConf {
.intConf
.createWithDefault(1)
+ val BINARY_COMPARISON_COMPATIBLE_WITH_HIVE =
+buildConf("spark.sql.binary.comparison.compatible.with.hive")
+ .doc("Whether compatible with Hive when binary comparison.")
--- End diff --
Binary comparison is just one of the implicit type casting cases.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139464467
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -925,6 +925,12 @@ object SQLConf {
.intConf
.createWithDefault(1)
+ val BINARY_COMPARISON_COMPATIBLE_WITH_HIVE =
+buildConf("spark.sql.binary.comparison.compatible.with.hive")
--- End diff --
-> `spark.sql.autoTypeCastingCompatibility`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r139464231
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -925,6 +925,12 @@ object SQLConf {
.intConf
.createWithDefault(1)
+ val BINARY_COMPARISON_COMPATIBLE_WITH_HIVE =
+buildConf("spark.sql.binary.comparison.compatible.with.hive")
+ .doc("Whether compatible with Hive when binary comparison.")
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
This has to be `false`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18853: [SPARK-21646][SQL] CommonType for binary comparis...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/18853#discussion_r137965884
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1966,7 +1966,7 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
test("SPARK-17913: compare long and string type column may return
confusing result") {
val df = Seq(123L -> "123", 19157170390056973L ->
"19157170390056971").toDF("i", "j")
-checkAnswer(df.select($"i" === $"j"), Row(true) :: Row(false) :: Nil)
+checkAnswer(df.select($"i" === $"j"), Row(true) :: Row(true) :: Nil)
--- End diff --
To compatible with Hive, MySQL and Oracle:
https://user-images.githubusercontent.com/5399861/30254791-f2e6cc28-96cf-11e7-817e-c837c42d7504.png;>
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
