[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321946772
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -861,7 +861,9 @@ object TypeCoercion {
// Skip nodes who's children have not been resolved yet.
case e if !e.childrenResolved => e
- case b @ BinaryOperator(left, right) if left.dataType != right.dataType
=>
+ // For DecimalType, it will be handled by another rule DecimalPrecision.
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321946764
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,18 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession {
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+withSQLConf("spark.sql.decimalOperations.allowPrecisionLoss" -> "false") {
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321763727
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -861,7 +861,8 @@ object TypeCoercion {
// Skip nodes who's children have not been resolved yet.
case e if !e.childrenResolved => e
- case b @ BinaryOperator(left, right) if left.dataType != right.dataType
=>
+ case b @ BinaryOperator(left, right) if
!left.dataType.isInstanceOf[DecimalType] &&
+ !right.dataType.isInstanceOf[DecimalType] && left.dataType !=
right.dataType =>
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321657464
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -861,7 +861,8 @@ object TypeCoercion {
// Skip nodes who's children have not been resolved yet.
case e if !e.childrenResolved => e
- case b @ BinaryOperator(left, right) if left.dataType != right.dataType
=>
+ case b @ BinaryOperator(left, right) if
!left.dataType.isInstanceOf[DecimalType] &&
+!right.dataType.isInstanceOf[DecimalType] && left.dataType !=
right.dataType =>
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321657417
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala
##
@@ -1576,6 +1576,20 @@ class TypeCoercionSuite extends AnalysisTest {
SpecifiedWindowFrame(RangeFrame, CurrentRow, UnboundedFollowing))
)
}
+
+ test("SPARK-29000 skip to handle decimals in ImplicitTypeCasts") {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321649874
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
##
@@ -862,17 +862,23 @@ object TypeCoercion {
case e if !e.childrenResolved => e
case b @ BinaryOperator(left, right) if left.dataType != right.dataType
=>
-findTightestCommonType(left.dataType, right.dataType).map { commonType
=>
- if (b.inputType.acceptsType(commonType)) {
-// If the expression accepts the tightest common type, cast to
that.
-val newLeft = if (left.dataType == commonType) left else
Cast(left, commonType)
-val newRight = if (right.dataType == commonType) right else
Cast(right, commonType)
-b.withNewChildren(Seq(newLeft, newRight))
- } else {
-// Otherwise, don't do anything with the expression.
-b
- }
-}.getOrElse(b) // If there is no applicable conversion, leave
expression unchanged.
+(left, right) match {
+ // Skip to handle decimals
+ case (l, r) if l.dataType.isInstanceOf[DecimalType] ||
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321649817
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,16 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession {
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+withSQLConf(("spark.sql.decimalOperations.allowPrecisionLoss", "false")) {
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321598873
##
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##
@@ -3180,6 +3180,15 @@ class SQLQuerySuite extends QueryTest with
SharedSparkSession {
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+val allow = sql("set spark.sql.decimalOperations.allowPrecisionLoss")
Review comment:
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321596864
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
##
@@ -1224,6 +1224,15 @@ class HiveQuerySuite extends HiveComparisonTest with
SQLTestUtils with BeforeAnd
}
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+val allow = sql("set spark.sql.decimalOperations.allowPrecisionLoss")
+val originValue = allow.collect().apply(0).apply(1)
+sql("set spark.sql.decimalOperations.allowPrecisionLoss=false")
+val df = sql("select case when 1=2 then 1 else
100. end * 1")
+QueryTest.checkAnswer(df, Array(Row(100)))
+sql(s"set spark.sql.decimalOperations.allowPrecisionLoss=$originValue")
+ }
Review comment:
I have added relative tests in `TypeCoericionSuite`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321596864
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
##
@@ -1224,6 +1224,15 @@ class HiveQuerySuite extends HiveComparisonTest with
SQLTestUtils with BeforeAnd
}
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+val allow = sql("set spark.sql.decimalOperations.allowPrecisionLoss")
+val originValue = allow.collect().apply(0).apply(1)
+sql("set spark.sql.decimalOperations.allowPrecisionLoss=false")
+val df = sql("select case when 1=2 then 1 else
100. end * 1")
+QueryTest.checkAnswer(df, Array(Row(100)))
+sql(s"set spark.sql.decimalOperations.allowPrecisionLoss=$originValue")
+ }
Review comment:
I have added relative tests in TypeCoericionSuite.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on a change in pull request #25701: [SPARK-29000][SQL] Decimal precision overflow when don't allow precision loss
turboFei commented on a change in pull request #25701: [SPARK-29000][SQL]
Decimal precision overflow when don't allow precision loss
URL: https://github.com/apache/spark/pull/25701#discussion_r321547661
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
##
@@ -1224,6 +1224,15 @@ class HiveQuerySuite extends HiveComparisonTest with
SQLTestUtils with BeforeAnd
}
}
}
+
+ test("SPARK-29000: arithmetic computation overflow when don't allow decimal
precision loss ") {
+val allow = sql("set spark.sql.decimalOperations.allowPrecisionLoss")
+val originValue = allow.collect().apply(0).apply(1)
+sql("set spark.sql.decimalOperations.allowPrecisionLoss=false")
+val df = sql("select case when 1=2 then 1 else
100. end * 1")
+QueryTest.checkAnswer(df, Array(Row(100)))
+sql(s"set spark.sql.decimalOperations.allowPrecisionLoss=$originValue")
+ }
Review comment:
I have moved it into SQLQuerySuite in package `sql`. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
