[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r293162657
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
For sql
```
create table t (d double);
insert into t values (10.0);
```
the 10.0 is decimal in Spark SQL parser.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292794102
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
We need to figure out if we can up-cast decimal to double/float.
If we can't, then maybe we can't continue this PR.
I have created a new PR for the upcasting:
https://github.com/apache/spark/pull/24849
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292794102
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
We need to figure out if we can up-cast decimal to double/float.
If we can't, then maybe we can't continue this PR.
I have created a new PR https://github.com/apache/spark/pull/24849
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292786952
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -128,8 +128,10 @@ object Cast {
case _ if from == to => true
case (from: NumericType, to: DecimalType) if to.isWiderThan(from) => true
case (from: DecimalType, to: NumericType) if from.isTighterThan(to) => true
-case (f, t) if legalNumericPrecedence(f, t) => true
+case (f: NumericType, t: NumericType) if legalNumericPrecedence(f, t) =>
true
+
case (DateType, TimestampType) => true
+case (NullType, _) => false
Review comment:
Here we can't know the nullability of the `to` type. We should consider it
is not nullable.
For the case you mentioned, it is handled in
https://github.com/apache/spark/pull/24806/files#diff-86e655772e8f7cab055d2c2451b52275R134.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292774456
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DataSourceV2AnalysisSuite.scala
##
@@ -290,15 +290,14 @@ abstract class DataSourceV2AnalysisSuite extends
AnalysisTest {
StructField("y", DoubleType))).toAttributes)
val query = TestRelation(StructType(Seq(
- StructField("x", DoubleType),
+ StructField("x", FloatType),
Review comment:
In this test case, the error message `"Cannot safely cast", "'x'",
"DoubleType to FloatType"` is gone after code changes.
So I think we should make the test case simpler. The nullability error in
the `x` column is enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806:
[WIP][SPARK-27856][SQL] Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r292770658
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala
##
@@ -89,6 +90,7 @@ case class DecimalType(precision: Int, scale: Int) extends
FractionalType {
(precision - scale) <= (dt.precision - dt.scale) && scale <= dt.scale
case dt: IntegralType =>
isTighterThan(DecimalType.forType(dt))
+// For DoubleType/FloatType, the value can be NaN, PositiveInfinity or
NegativeInfinity.
Review comment:
> For isTighterThan, I think it's safe to cast decimal to float/double if
the precision doesn't exceed?
Yes, it is. I was about to push the commit to fix tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
