This is an automated email from the ASF dual-hosted git repository.
ulysses-you pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 74796290ac [GLUTEN-12260][VL] Fix CheckOverflowTransformer using wrong
child type for cast decision in Spark-33 (#12261)
74796290ac is described below
commit 74796290acb9d100a3bf7499de78399164c25316
Author: Shilong Duan <[email protected]>
AuthorDate: Tue Jun 23 10:11:00 2026 +0800
[GLUTEN-12260][VL] Fix CheckOverflowTransformer using wrong child type for
cast decision in Spark-33 (#12261)
Fix: https://github.com/apache/gluten/issues/12260
## What changes are proposed in this pull request?
`CheckOverflowTransformer` reads `original.child.dataType` to decide
whether to insert a cast. For `BinaryArithmetic`, Spark's `.dataType` returns
`left.dataType` rather than the arithmetic result type. After child
transformers apply rescale optimizations, the actual output type may differ
from the Spark-declared type, and the cast is wrongly skipped.
The resulting substrait plan has decimal types that mismatch function
signatures. Velox's SimpleFunction validation rejects it, and
`ColumnarPartialProjectRule` falls the entire Project back to JVM. Result is
correct (via fallback) but native acceleration is lost.
### Reproducer
```sql
CREATE TABLE t1 (val BIGINT) USING parquet;
CREATE TABLE t2 (val BIGINT) USING parquet;
INSERT INTO t1 VALUES (200);
INSERT INTO t2 VALUES (100), (100), (100), (100), (100);
SELECT
a.val,
(a.val - COALESCE(SUM(b.val), 0) / 5.0)
/ (COALESCE(SUM(b.val), 0) / 5.0) AS growth_rate
FROM t1 a CROSS JOIN t2 b
GROUP BY a.val;
```
### Root cause:
https://github.com/apache/gluten/blob/fc90a7933afaf5518ad20a60a5d79d482cea5ef1/gluten-substrait/src/main/scala/org/apache/gluten/expression/UnaryExpressionTransformer.scala#L90
this read the Spark expression's declared type instead of the transformer's
actual output type.
### Fix
```diff
- original.child.dataType,
+ child.dataType,
```
## How was this patch tested?
**Before fix** — Project falls back to JVM:
```
== Final Plan ==
* Project (17) ← JVM, codegen id=3
+- VeloxColumnarToRow (16) ← extra C2R
conversion
+- ^ RegularHashAggregateExecTransformer (14)
+- ^ VeloxBroadcastNestedLoopJoinExecTransformer (13)
:- ^ InputIteratorTransformer (7)
: +- BroadcastQueryStage (5)
: +- ColumnarBroadcastExchange (4)
: +- RowToVeloxColumnar (3)
: +- * ColumnarToRow (2)
: +- BatchScan (1)
+- ^ InputIteratorTransformer (12)
+- RowToVeloxColumnar (10)
+- * ColumnarToRow (9)
+- BatchScan (8)
```
**After fix** — Project runs natively in Velox:
```
== Final Plan ==
VeloxColumnarToRow (17)
+- ^ ProjectExecTransformer (15) ← native Velox
Project
+- ^ RegularHashAggregateExecTransformer (14)
+- ^ VeloxBroadcastNestedLoopJoinExecTransformer (13)
:- ^ InputIteratorTransformer (7)
: +- BroadcastQueryStage (5)
: +- ColumnarBroadcastExchange (4)
: +- RowToVeloxColumnar (3)
: +- * ColumnarToRow (2)
: +- BatchScan (1)
+- ^ InputIteratorTransformer (12)
+- RowToVeloxColumnar (10)
+- * ColumnarToRow (9)
+- BatchScan (8)
```
Key differences:
- Node (15) changes from `Project` (JVM, `*` = codegen) to
`ProjectExecTransformer` (Velox native, `^` = transformer)
- `VeloxColumnarToRow` moves from **before** Project (forced conversion
to feed JVM) to **after** Project (deferred until output)
- Aggregate→Project pipeline stays in Velox without breaking
---
.../expression/UnaryExpressionTransformer.scala | 2 +-
.../GlutenCheckOverflowTransformerSuite.scala | 70 ++++++++++++++++++++++
2 files changed, 71 insertions(+), 1 deletion(-)
diff --git
a/gluten-substrait/src/main/scala/org/apache/gluten/expression/UnaryExpressionTransformer.scala
b/gluten-substrait/src/main/scala/org/apache/gluten/expression/UnaryExpressionTransformer.scala
index b762ec95b6..1c0faf599b 100644
---
a/gluten-substrait/src/main/scala/org/apache/gluten/expression/UnaryExpressionTransformer.scala
+++
b/gluten-substrait/src/main/scala/org/apache/gluten/expression/UnaryExpressionTransformer.scala
@@ -87,7 +87,7 @@ case class CheckOverflowTransformer(
context,
substraitExprName,
child.doTransform(context),
- original.child.dataType,
+ child.dataType,
original.dataType,
original.nullable,
original.nullOnOverflow)
diff --git
a/gluten-ut/test/src/test/scala/org/apache/gluten/expressions/GlutenCheckOverflowTransformerSuite.scala
b/gluten-ut/test/src/test/scala/org/apache/gluten/expressions/GlutenCheckOverflowTransformerSuite.scala
new file mode 100644
index 0000000000..d97dea2f38
--- /dev/null
+++
b/gluten-ut/test/src/test/scala/org/apache/gluten/expressions/GlutenCheckOverflowTransformerSuite.scala
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.expressions
+
+import org.apache.gluten.expression._
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.utils.BackendTestUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.GlutenQueryTest
+import org.apache.spark.sql.catalyst.expressions.{CheckOverflow, Literal}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{Decimal, DecimalType}
+
+class GlutenCheckOverflowTransformerSuite extends GlutenQueryTest with
SharedSparkSession {
+
+ override protected def sparkConf: SparkConf = {
+ super.sparkConf
+ .set("spark.plugins", "org.apache.gluten.GlutenPlugin")
+ .set("spark.default.parallelism", "1")
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "1024MB")
+ .set("spark.ui.enabled", "false")
+ }
+
+ testWithSpecifiedSparkVersion(
+ "CheckOverflow transformer casts transformed child type",
+ "3.3",
+ "3.4",
+ "3.5",
+ "4.0",
+ "4.1") {
+ assume(BackendTestUtils.isVeloxBackendLoaded())
+
+ val targetType = DecimalType(38, 17)
+ val transformedChildType = DecimalType(38, 18)
+ val original = CheckOverflow(
+ Literal(Decimal(0, targetType.precision, targetType.scale), targetType),
+ targetType,
+ nullOnOverflow = true)
+ val child = LiteralTransformer(
+ Literal(
+ Decimal(0, transformedChildType.precision, transformedChildType.scale),
+ transformedChildType))
+ assert(original.child.dataType != child.dataType)
+
+ val transformedNode =
+ CheckOverflowTransformer(ExpressionNames.CHECK_OVERFLOW, child, original)
+ .doTransform(new SubstraitContext)
+ .toProtobuf
+ assert(transformedNode.hasCast)
+ val castType = transformedNode.getCast.getType.getDecimal
+ assert(castType.getPrecision == targetType.precision)
+ assert(castType.getScale == targetType.scale)
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
