This is an automated email from the ASF dual-hosted git repository. yumwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new c20af53 [SPARK-36373][SQL] DecimalPrecision only add necessary cast c20af53 is described below commit c20af535803a7250fef047c2bf0fe30be242369d Author: Yuming Wang <yumw...@ebay.com> AuthorDate: Tue Aug 3 08:12:54 2021 +0800 [SPARK-36373][SQL] DecimalPrecision only add necessary cast ### What changes were proposed in this pull request? This pr makes `DecimalPrecision` only add necessary cast similar to [`ImplicitTypeCasts`](https://github.com/apache/spark/blob/96c2919988ddf78d104103876d8d8221e8145baa/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L675-L678). For example: ``` EqualTo(AttributeReference("d1", DecimalType(5, 2))(), AttributeReference("d2", DecimalType(2, 1))()) ``` It will add a useless cast to _d1_: ``` (cast(d1#6 as decimal(5,2)) = cast(d2#7 as decimal(5,2))) ``` ### Why are the changes needed? 1. Avoid adding unnecessary cast. Although it will be removed by `SimplifyCasts` later. 2. I'm trying to add an extended rule similar to `PullOutGroupingExpressions`. The current behavior will introduce additional alias. For example: `cast(d1 as decimal(5,2)) as cast_d1`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. Closes #33602 from wangyum/SPARK-36373. Authored-by: Yuming Wang <yumw...@ebay.com> Signed-off-by: Yuming Wang <yumw...@ebay.com> --- .../org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala | 4 +++- .../apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala | 3 +++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala index bf128cd..b112641 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala @@ -204,7 +204,9 @@ object DecimalPrecision extends TypeCoercionRule { case b @ BinaryComparison(e1 @ DecimalType.Expression(p1, s1), e2 @ DecimalType.Expression(p2, s2)) if p1 != p2 || s1 != s2 => val resultType = widerDecimalType(p1, s1, p2, s2) - b.makeCopy(Array(Cast(e1, resultType), Cast(e2, resultType))) + val newE1 = if (e1.dataType == resultType) e1 else Cast(e1, resultType) + val newE2 = if (e2.dataType == resultType) e2 else Cast(e2, resultType) + b.makeCopy(Array(newE1, newE2)) } /** diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala index 834f38a..698d2d5 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala @@ -59,6 +59,9 @@ class DecimalPrecisionSuite extends AnalysisTest with BeforeAndAfter { val comparison = analyzer.execute(plan).collect { case Project(Alias(e: BinaryComparison, _) :: Nil, _) => e }.head + // Only add necessary cast. + assert(comparison.left.children.forall(_.dataType !== expectedType)) + assert(comparison.right.children.forall(_.dataType !== expectedType)) assert(comparison.left.dataType === expectedType) assert(comparison.right.dataType === expectedType) } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org