pedrumj2 opened a new pull request, #13604:
URL: https://github.com/apache/gluten/pull/13604

   ## What changes are proposed in this pull request?
   
   When ANSI mode is on, Spark's 
[try_add](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_add), 
[try_subtract](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_subtract)
 and 
[try_multiply](https://spark.apache.org/docs/3.5.6/api/sql/index.html#try_multiply)
 on integers throw an error when computing one of their inputs fails. This PR 
makes the native path follow the same pattern:
   
   | Query with ANSI mode on, where `z = 0` | Native before this PR | Native 
with this PR | Spark |
   |---|---|---|---|
   | `try_add(CAST(a / z AS INT), 1)` | `NULL` | throws division by zero | 
throws `DIVIDE_BY_ZERO` |
   
   The native path now sends Velox a different function for each version of try:
   
   | Spark function | Sent to Velox before this PR | Sent to Velox with this PR 
|
   |---|---|---|
   | `try_add(left, right)` | `try(checked_add(left, right))` | `try_add(left, 
right)` |
   | `try_subtract(left, right)` | `try(checked_subtract(left, right))` | 
`try_subtract(left, right)` |
   | `try_multiply(left, right)` | `try(checked_multiply(left, right))` | 
`try_multiply(left, right)` |
   
   Velox's 
[`try`](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/expression/TryExpr.cpp#L21-L48)
 turns every error inside it into `NULL`.  
[facebookincubator/velox#19463](https://github.com/facebookincubator/velox/pull/19463)
 added Velox's own `try_add`, `try_subtract` and `try_multiply`, which return 
`NULL` only when the operation overflows. 
   
   
   This repo's Velox, the 
[dft-2026_11_06](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/registration/RegisterMath.cpp#L176-L178)
 tag, has these operators.
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout cdf5292884895126825e4349938f2da3959fc812
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write the input table and the query:
   
   ```bash
   cat > write_input.sql <<'EOF'
   CREATE TABLE nums USING parquet AS SELECT 1 AS a, 0 AS z;
   EOF
   cat > query.sql <<'EOF'
   EXPLAIN SELECT try_add(CAST(a / z AS INT), 1) FROM nums;
   SELECT try_add(CAST(a / z AS INT), 1) FROM nums;
   EOF
   ```
   
   5. Write the table with plain Spark, then query it with Gluten, with ANSI 
mode on and Gluten's ANSI fallback off:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     rm -rf metastore_db spark-warehouse
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f 
write_input.sql 2>/dev/null
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       --conf spark.sql.ansi.enabled=true \
       --conf spark.gluten.sql.ansiFallback.enabled=false \
       -f query.sql 2>err.log
     grep -m 1 -E "DIVIDE_BY_ZERO|division by zero" err.log
   '
   ```
   
   On the base commit, the query runs natively and returns `NULL`. The grep 
finds no error:
   
   ```
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [(cast((cast(a#5 as double) / cast(z#6 as 
double)) as int) + 1) AS try_add(CAST((a / z) AS INT), 1)#7]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.nums[a#5,z#6] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:int,z:int> NativeFilters: []
   
   
   NULL
   ```
   
   On this pull request's branch, rerunning steps 2 and 5 shows the same native 
plan. The query now fails with Velox's error:
   
   ```
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [(cast((cast(a#5 as double) / cast(z#6 as 
double)) as int) + 1) AS try_add(CAST((a / z) AS INT), 1)#7]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.nums[a#5,z#6] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:int,z:int> NativeFilters: []
   
   
   Reason: Reason: (0 vs. 0) division by zero
   ```
   
   Spark without Gluten fails the same way. To run it, remove every option from 
step 5's second command except `--master`, [`--conf 
spark.sql.ansi.enabled=true`](https://spark.apache.org/docs/3.5.6/sql-ref-ansi-compliance.html)
 and `-f query.sql`:
   
   ```
   == Physical Plan ==
   *(1) Project [(cast((cast(a#5 as double) / cast(z#6 as double)) as int) + 1) 
AS try_add(CAST((a / z) AS INT), 1)#7]
   +- *(1) ColumnarToRow
      +- FileScan parquet spark_catalog.default.nums[a#5,z#6] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/nums], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<a:int,z:int>
   
   
   org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by 
zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If 
necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
   ```
   
   ### Automated tests
   
   `integral try_add, try_subtract and try_multiply` in 
[ArithmeticAnsiValidateSuite](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/test/scala/org/apache/gluten/functions/ArithmeticAnsiValidateSuite.scala)
 checks on Velox that an overflow returns `NULL` and that an input that divides 
by zero throws.
   
   On the base commit, the new test fails:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl 
backends-velox -Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.ArithmeticAnsiValidateSuite 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   - integral try_add, try_subtract and try_multiply *** FAILED ***
     Expected exception org.apache.spark.SparkException to be thrown, but no 
exception was thrown (ArithmeticAnsiValidateSuite.scala:204)
   …
   Tests: succeeded 12, failed 1, canceled 0, ignored 0, pending 0
   ```
   
   With this change, the same command passes:
   
   ```
   …
   - integral try_add, try_subtract and try_multiply
   …
   Tests: succeeded 13, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl 
backends-velox -Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.MathFunctionsValidateSuite 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   MathFunctionsValidateSuiteAnsiOn:
   …
   - try_add
   …
   - try_divide
   …
   - try_multiply
   …
   - try_subtract
   …
   MathFunctionsValidateSuite:
   …
   - try_add
   …
   - try_divide
   …
   - try_multiply
   …
   - try_subtract
   …
   Tests: succeeded 51, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   ```bash
   ./build/mvn -o -B test -Pspark-ut -Pbackends-velox -Pspark-3.5 -pl 
gluten-ut/spark35 -Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.catalyst.expressions.GlutenTryEvalSuite,org.apache.spark.sql.catalyst.expressions.GlutenArithmeticExpressionSuite,org.apache.spark.sql.GlutenMathFunctionsSuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   GlutenTryEvalSuite:
   …
   GlutenArithmeticExpressionSuite:
   …
   GlutenMathFunctionsSuite:
   …
   Tests: succeeded 99, failed 0, canceled 0, ignored 1, pending 0
   All tests passed.
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to