pedrumj2 opened a new pull request, #13600: URL: https://github.com/apache/gluten/pull/13600
## What changes are proposed in this pull request? Native [transform](https://spark.apache.org/docs/3.5.6/api/sql/index.html#transform) and [filter](https://spark.apache.org/docs/3.5.6/api/sql/index.html#filter) now run lambdas that take the element index, such as `transform(arr, (x, i) -> x + i)`, instead of falling back to the JVM. [genArrayTransformTransformer](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L269-L271) and [genArrayFilterTransformer](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L219-L221) throw for a lambda with two arguments. [facebookincubator/velox#16211](https://github.com/facebookincubator/velox/pull/16211) and [facebookincubator/velox#16334](https://github.com/facebookincubator/velox/pull/16334) added the index variants of Velox's Spark `transform` and `filter`, this repo's Velox has carried them since apache/gluten#11624 and apache/gluten#11717, and the [dft-2026_11_06](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/registration/RegisterArray.cpp#L255-L258) tag registers both. This P R removes the two throws. A projection that holds such a lambda now stays native, together with every other expression in it. ## How was this patch tested? ### Reproduction 1. Clone the repo and check out the base commit: ```bash git clone https://github.com/apache/gluten.git cd gluten git checkout cdf5292884895126825e4349938f2da3959fc812 ``` 2. Build the native libraries and the bundle jar: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable cd /work/gluten ./dev/builddeps-veloxbe.sh --run_setup_script=OFF ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests ' ``` 3. Fetch Spark 3.5.5: ```bash curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz ``` 4. Write the input table and the query: ```bash cat > write_input.sql <<'EOF' CREATE TABLE t USING parquet AS SELECT array(0, 2, 3) AS arr; EOF cat > query.sql <<'EOF' EXPLAIN SELECT transform(arr, (x, i) -> x + i), filter(arr, (x, i) -> x > i) FROM t; SELECT transform(arr, (x, i) -> x + i), filter(arr, (x, i) -> x > i) FROM t; EOF ``` 5. Write the table with plain Spark, then query it with Gluten: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' cd /work/gluten rm -rf metastore_db spark-warehouse spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f write_input.sql 2>/dev/null BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ -f query.sql 2>err.log grep -m 1 "index argument" err.log ' ``` On the base commit, the projection runs on the JVM, and the grep finds Gluten's reason: ``` == Physical Plan == Project [transform(arr#5, lambdafunction((lambda x#6 + lambda i#7), lambda x#6, lambda i#7, false)) AS transform(arr, lambdafunction((namedlambdavariable() + namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#11] +- VeloxColumnarToRow +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[arr#5] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<arr:array<int>> NativeFilters: [] [0,3,5] [2,3] - Validation failed with exception from: ProjectExecTransformer, reason: transform on array with lambda using index argument is not supported yet ``` On this pull request's branch, with the bundle jar rebuilt, step 5 returns the same result from a native plan, and the grep finds nothing: ``` == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [transform(arr#5, lambdafunction((lambda x#6 + lambda i#7), lambda x#6, lambda i#7, false)) AS transform(arr, lambdafunction((namedlambdavariable() + namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#11] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[arr#5] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<arr:array<int>> NativeFilters: [] [0,3,5] [2,3] ``` Spark without Gluten, which is step 5's second command without the `--jars`, `--driver-class-path` and `--conf` options, returns the same result: ``` == Physical Plan == Project [transform(arr#5, lambdafunction((lambda x#6 + lambda i#7), lambda x#6, lambda i#7, false)) AS transform(arr, lambdafunction((namedlambdavariable() + namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#10, filter(arr#5, lambdafunction((lambda x#8 > lambda i#9), lambda x#8, lambda i#9, false)) AS filter(arr, lambdafunction((namedlambdavariable() > namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#11] +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.t[arr#5] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<arr:array<int>> [0,3,5] [2,3] ``` ### Automated tests The `filter` and `transform` tests in [ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1058-L1103) each gain an index lambda, over null elements and null and empty arrays, checked to run natively. With this pull request's tests on the base commit's code, both tests fail on the new cases: ```bash ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite @filter,org.apache.gluten.functions.ScalarFunctionsValidateSuite @transform" ``` ``` - filter *** FAILED *** executedPlan.exists(((plan: org.apache.spark.sql.execution.SparkPlan) => tag.runtimeClass.isInstance(plan))) was false Expect ProjectExecTransformer exists in executedPlan: Project [filter(value#144, lambdafunction((lambda i#183 < (size(value#144, true) - 1)), lambda x#182, lambda i#183, false)) AS res#181] +- VeloxColumnarToRow … - transform *** FAILED *** executedPlan.exists(((plan: org.apache.spark.sql.execution.SparkPlan) => tag.runtimeClass.isInstance(plan))) was false Expect ProjectExecTransformer exists in executedPlan: Project [transform(arr#332, lambdafunction((lambda x#351 + lambda i#352), lambda x#351, lambda i#352, false)) AS transform(arr, lambdafunction((namedlambdavariable() + namedlambdavariable()), namedlambdavariable(), namedlambdavariable()))#353] +- VeloxColumnarToRow … Tests: succeeded 0, failed 2, canceled 0, ignored 0, pending 0 *** 2 TESTS FAILED *** ``` With this change: ```bash ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite ``` ``` ScalarFunctionsValidateSuite: … - filter … - transform … Tests: succeeded 90, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` ```bash ./build/mvn -o -B test -Pspark-ut -Pbackends-velox -Pspark-3.5 -pl gluten-ut/spark35 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.spark.sql.catalyst.expressions.GlutenHigherOrderFunctionsSuite,org.apache.spark.sql.GlutenDataFrameFunctionsSuite ``` ``` GlutenHigherOrderFunctionsSuite: … - ArrayTransform … - ArrayFilter … GlutenDataFrameFunctionsSuite: … - transform function - array for primitive type not containing null … - transform function - array for primitive type containing null … - transform function - array for non-primitive type … - filter function - index argument … Tests: succeeded 158, failed 0, canceled 0, ignored 4, pending 0 All tests passed. ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) <!-- Thank you for submitting a pull request! Here are some tips: 1. For first-time contributors, please read our contributing guide: https://github.com/apache/gluten/blob/main/CONTRIBUTING.md 2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work. 3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the Velox or ClickHouse backend, respectively. 4. If the PR is not ready for review, please mark it as a draft. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
