luis4a0 commented on code in PR #13103:
URL: https://github.com/apache/gluten/pull/13103#discussion_r4536974852
##########
cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc:
##########
@@ -152,6 +152,19 @@ bool SubstraitToVeloxPlanValidator::validateRound(
return false;
}
+ const auto& valueType = params[0]->type();
+ if (valueType->isDecimal()) {
+ const auto& scale = arguments[1].value().literal();
Review Comment:
This is the same private-call invariant: `validateScalarFunction` creates
`params` in a 1:1 loop over `scalarFunction.arguments()` and returns before
calling `validateRound` if an argument cannot be validated. The converter
contract supplies one non-null expression per argument. Thus two Substrait
arguments imply at least two converted parameters; no independently incomplete
vector can reach this function.
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/DecimalRoundTransformer.scala:
##########
@@ -20,15 +20,16 @@ import org.apache.gluten.backendsapi.BackendsApiManager
import org.apache.gluten.exception.GlutenNotSupportException
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.types.{DataType, DecimalType}
+import org.apache.spark.sql.types.{DataType, DecimalType, IntegerType}
case class DecimalRoundTransformer(
substraitExprName: String,
child: ExpressionTransformer,
original: Round)
extends BinaryExpressionTransformer {
- val toScale: Int = original.scale.eval(EmptyRow).asInstanceOf[Int]
+ private val scaleValue = original.scale.eval(EmptyRow)
+ val toScale: Int = if (scaleValue == null) 0 else
scaleValue.asInstanceOf[Int]
Review Comment:
An invalid foldable scale is an error in Spark as well, not a case where
fallback makes the query succeed. Spark `RoundBase` evaluates
`scale.eval(EmptyRow)` into `_scale`, and decimal result typing reads `_scale`
during planning. Converting that error into an unsupported-expression fallback
would only defer the same Spark error and change its timing. The API regression
intentionally verifies that the invalid cast throws while the generic Scala
validator does not impose backend-specific eligibility. No catch is needed here.
##########
gluten-ut/common/src/test/scala/org/apache/spark/sql/GlutenDecimalRoundSuiteBase.scala:
##########
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql
+
+import org.apache.gluten.config.GlutenConfig
+import org.apache.gluten.execution.ProjectExecTransformer
+
+import org.apache.spark.sql.catalyst.expressions.{Literal, Round}
+import org.apache.spark.sql.catalyst.optimizer.{ConstantFolding,
NullPropagation}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+import scala.collection.JavaConverters._
+
+abstract class GlutenDecimalRoundSuiteBase extends GlutenSQLTestsTrait {
+ private def withInput(dataType: DecimalType, values: Seq[String])(f: =>
Unit): Unit = {
+ val schema = new StructType().add("value", dataType)
+ val rows = values.map(value => Row(new java.math.BigDecimal(value))) :+
Row(null)
+ withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+ GlutenConfig.GLUTEN_ANSI_FALLBACK_ENABLED.key -> "false") {
+ withTempPath {
+ path =>
+ spark.createDataFrame(rows.asJava,
schema).coalesce(1).write.parquet(path.toString)
+ withTempView("decimal_round_input") {
+
spark.read.parquet(path.toString).createOrReplaceTempView("decimal_round_input")
+ f
+ }
+ }
+ }
+ }
+
+ private def assertExecution(df: DataFrame, native: Boolean): Unit = {
+ val plan = df.queryExecution.executedPlan
+ val locations = plan.collect {
+ case node if node.expressions.exists(_.exists(_.isInstanceOf[Round])) =>
+ node.isInstanceOf[ProjectExecTransformer]
+ }
+ assert(
+ locations.nonEmpty && locations.forall(_ == native),
+ s"Unexpected ROUND execution:\n$plan")
+ }
+
+ private def checkRound(scale: String, native: Boolean = true): Unit = {
+ val query = s"SELECT round(value, $scale) FROM decimal_round_input"
+ withSQLConf(GlutenConfig.GLUTEN_ENABLED.key -> "false") {
+ val vanilla = sql(query)
+ val expectedSchema = vanilla.schema
+ val expected = vanilla.collect().toSeq
+ withSQLConf(GlutenConfig.GLUTEN_ENABLED.key -> "true") {
+ val df = sql(query)
+ assertExecution(df, native)
+ assert(df.schema == expectedSchema)
+ checkAnswer(df, expected)
+ }
+ }
+ }
+
+ testGluten("decimal ROUND preserves schemas and HALF_UP results across
storage boundaries") {
+ val cases = Seq(
+ (DecimalType(3, 1), Seq("2.5", "-2.5", "0")),
+ (DecimalType(18, 0), Seq("999999999999999999", "-999999999999999999")),
+ (DecimalType(19, 0), Seq("9999999999999999999", "-9999999999999999999")),
+ (DecimalType(38, 1), Seq("9" * 37 + ".9", "-" + "9" * 37 + ".9")),
+ (DecimalType(38, 38), Seq("0.6", "-0.6", "0.5", "-0.5"))
+ )
+ for ((dataType, values) <- cases) {
+ withInput(dataType, values) {
+ for (ansi <- Seq("false", "true")) {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
+ Seq(-400, -39, -1, 0, 1, 38, 400).foreach(scale =>
checkRound(scale.toString))
+ checkRound("-1 + 0")
+ }
+ }
+ }
+ }
+ }
+
+ testGluten("decimal ROUND handles the 38-digit HALF_UP tie and precision
overflow") {
+ withInput(DecimalType(38, 0), Seq("4" + "9" * 37, "-4" + "9" * 37)) {
+ checkRound("-38")
+ }
+ for ((value, scale) <- Seq(("9" * 38, -1), ("5" + "0" * 37, -38), ("-5" +
"0" * 37, -38))) {
+ withInput(DecimalType(38, 0), Seq(value)) {
+ for (ansi <- Seq("false", "true")) {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
+ val query = s"SELECT round(value, $scale) FROM decimal_round_input"
+ withSQLConf(GlutenConfig.GLUTEN_ENABLED.key -> "false") {
+ val error = intercept[Exception](sql(query).collect())
+
assert(Iterator.iterate[Throwable](error)(_.getCause).takeWhile(_ !=
null).exists {
+ cause =>
+ Option(cause.getMessage).exists {
+ message =>
+ message.contains("NUMERIC_VALUE_OUT_OF_RANGE") ||
+
message.contains("DECIMAL_PRECISION_EXCEEDS_MAX_PRECISION")
+ }
+ })
+ }
+ val df = sql(query)
+ assertExecution(df, native = true)
+ val error = intercept[Exception](df.collect())
+ assert(Iterator.iterate[Throwable](error)(_.getCause).takeWhile(_
!= null).exists {
+ cause => Option(cause.getMessage).exists(_.contains("Decimal
overflow"))
+ })
Review Comment:
There is no more specific stable Spark error class exposed through the JNI
`GlutenException`; the generic native code is `INVALID_ARGUMENT`, which would
match many unrelated failures. The message token is the registered native
function contract: `DecimalRound.cpp` emits `Status::UserError("Decimal
overflow in {}.", name_)`, and Velox’s own `DecimalRoundTest` asserts `"Decimal
overflow in decimal_round."`. Keeping the focused substring verifies that this
exact overflow path was reached without coupling the Gluten test to the full
wrapper/stack-trace text.
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/DecimalRoundTransformer.scala:
##########
@@ -43,5 +44,6 @@ case class DecimalRoundTransformer(
}
override def left: ExpressionTransformer = child
- override def right: ExpressionTransformer = LiteralTransformer(toScale)
+ override def right: ExpressionTransformer =
+ LiteralTransformer(Literal.create(scaleValue, IntegerType))
Review Comment:
This eager evaluation matches Spark’s own `RoundBase` contract. Spark stores
`scale.eval(EmptyRow)` in `scaleV`/`_scale`, and decimal `dataType` depends on
`_scale`; a foldable ANSI cast failure therefore surfaces during Spark planning
too. Falling back cannot produce a successful Spark plan for that input and
would only move the same error. The current behavior is intentional.
##########
cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc:
##########
@@ -141,7 +141,7 @@ bool SubstraitToVeloxPlanValidator::flattenDualLevel(const
TypePtr& type, std::v
bool SubstraitToVeloxPlanValidator::validateRound(
const ::substrait::Expression::ScalarFunction& scalarFunction,
- const RowTypePtr& inputType) {
+ const std::vector<core::TypedExprPtr>& params) {
Review Comment:
Verified against the private caller contract. `validateScalarFunction`
iterates the Substrait arguments and appends exactly one converted, non-null
expression to `params` for each argument; if nested validation fails, it
returns before dispatching to `validateRound`. Therefore `params.size() ==
arguments.size()` here, and the existing `arguments.size() < 2` check
guarantees `params[0]` exists. The header documents this invariant, so an
additional size check would only duplicate an unreachable condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]