pedrumj2 opened a new pull request, #13603:
URL: https://github.com/apache/gluten/pull/13603

   ## What changes are proposed in this pull request?
   
   Native 
[filter](https://spark.apache.org/docs/3.5.6/api/sql/index.html#filter) now 
returns Spark's result when its lambda applies `IS NOT NULL` to a struct field, 
an outer column, or anything other than the element itself; only `filter(arr, x 
-> x IS NOT NULL)` still runs as 
[array_compact](https://spark.apache.org/docs/3.5.6/api/sql/index.html#array_compact).
   
   apache/gluten#9349 made 
[genArrayFilterTransformer](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L215-L218)
 run a `filter` as `array_compact` whenever the lambda body is 
[IsNotNull](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala#L380),
 whatever that `IsNotNull` tests.
   
   [Spark's 
array_compact](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L5050-L5052)
 removes null elements, and Spark defines it as a `filter` whose lambda tests 
the element itself. So these two queries return the same rows, and only this 
`filter` can run as `array_compact`:
   
   ```sql
   SELECT array_compact(arr) FROM t;
   SELECT filter(arr, x -> x IS NOT NULL) FROM t;
   ```
   
   A filter that tests a struct field or another column also runs as 
`array_compact`, and returns the wrong rows. For an `arr` of 
`[{"a":1},{"a":null}]` and a `c` of `NULL`:
   
   | Query | Spark | Gluten before this PR |
   | --- | --- | --- |
   | `filter(arr, x -> x IS NOT NULL)` | `[{"a":1},{"a":null}]` | 
`[{"a":1},{"a":null}]` |
   | `filter(arr, x -> x.a IS NOT NULL)` | `[{"a":1}]` | `[{"a":1},{"a":null}]` 
|
   | `filter(arr, x -> c IS NOT NULL)` | `[]` | `[{"a":1},{"a":null}]` |
   
   This PR runs a `filter` as `array_compact` only when its `IS NOT NULL` tests 
the lambda variable itself, in the Velox backend and in the same rewrite in 
[BoltSparkPlanExecApi](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-bolt/src/main/scala/org/apache/gluten/backendsapi/bolt/BoltSparkPlanExecApi.scala#L194).
 The second and third queries then run as `filter` and return Spark's rows, 
still natively.
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout cdf5292884895126825e4349938f2da3959fc812
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write the input table and the query:
   
   ```bash
   cat > write_input.sql <<'EOF'
   CREATE TABLE t USING parquet AS
   SELECT 1 AS id, array(named_struct('a', 1), named_struct('a', CAST(NULL AS 
INT))) AS arr, CAST(NULL AS INT) AS c
   UNION ALL
   SELECT 2, array(named_struct('a', 2), CAST(NULL AS STRUCT<a: INT>)), 3;
   EOF
   cat > query.sql <<'EOF'
   SELECT id, filter(arr, x -> x IS NOT NULL), filter(arr, x -> x.a IS NOT 
NULL), filter(arr, x -> c IS NOT NULL) FROM t ORDER BY id;
   EXPLAIN SELECT filter(arr, x -> x IS NOT NULL), filter(arr, x -> x.a IS NOT 
NULL), filter(arr, x -> c IS NOT NULL) FROM t;
   EOF
   ```
   
   5. Write the table with plain Spark, then query it with Gluten:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     rm -rf metastore_db spark-warehouse
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f 
write_input.sql 2>/dev/null
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       -f query.sql 2>/dev/null
   '
   ```
   
   On the base commit, all three filters keep every non-null element, in the 
native plan:
   
   ```
   1    [{"a":1},{"a":null}]    [{"a":1},{"a":null}]    [{"a":1},{"a":null}]
   2    [{"a":2}]       [{"a":2}]       [{"a":2}]
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [filter(arr#1, 
lambdafunction(isnotnull(lambda x#19), lambda x#19, false)) AS filter(arr, 
lambdafunction((namedlambdavariable() IS NOT NULL), namedlambdavariable()))#22, 
filter(arr#1, lambdafunction(isnotnull(lambda x#20.a), lambda x#20, false)) AS 
filter(arr, lambdafunction((namedlambdavariable().a IS NOT NULL), 
namedlambdavariable()))#23, filter(arr#1, lambdafunction(isnotnull(c#2), lambda 
x#21, false)) AS filter(arr, lambdafunction((c IS NOT NULL), 
namedlambdavariable()))#24]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[arr#1,c#2] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<arr:array<struct<a:int>>,c:int> 
NativeFilters: []
   ```
   
   On this pull request's branch, rerunning steps 2 and 5 returns Spark's rows, 
still in the native plan:
   
   ```
   1    [{"a":1},{"a":null}]    [{"a":1}]       []
   2    [{"a":2}]       [{"a":2}]       [{"a":2},null]
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [filter(arr#1, 
lambdafunction(isnotnull(lambda x#19), lambda x#19, false)) AS filter(arr, 
lambdafunction((namedlambdavariable() IS NOT NULL), namedlambdavariable()))#22, 
filter(arr#1, lambdafunction(isnotnull(lambda x#20.a), lambda x#20, false)) AS 
filter(arr, lambdafunction((namedlambdavariable().a IS NOT NULL), 
namedlambdavariable()))#23, filter(arr#1, lambdafunction(isnotnull(c#2), lambda 
x#21, false)) AS filter(arr, lambdafunction((c IS NOT NULL), 
namedlambdavariable()))#24]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[arr#1,c#2] Batched: true, DataFilters: [], Format: 
Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<arr:array<struct<a:int>>,c:int> 
NativeFilters: []
   ```
   
   Spark without Gluten, which is step 5's second command without the `--jars`, 
`--driver-class-path` and `--conf` options, returns the same rows:
   
   ```
   1    [{"a":1},{"a":null}]    [{"a":1}]       []
   2    [{"a":2}]       [{"a":2}]       [{"a":2},null]
   == Physical Plan ==
   Project [filter(arr#1, lambdafunction(isnotnull(lambda x#19), lambda x#19, 
false)) AS filter(arr, lambdafunction((namedlambdavariable() IS NOT NULL), 
namedlambdavariable()))#22, filter(arr#1, lambdafunction(isnotnull(lambda 
x#20.a), lambda x#20, false)) AS filter(arr, 
lambdafunction((namedlambdavariable().a IS NOT NULL), 
namedlambdavariable()))#23, filter(arr#1, lambdafunction(isnotnull(c#2), lambda 
x#21, false)) AS filter(arr, lambdafunction((c IS NOT NULL), 
namedlambdavariable()))#24]
   +- *(1) ColumnarToRow
      +- FileScan parquet spark_catalog.default.t[arr#1,c#2] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<arr:array<struct<a:int>>,c:int>
   ```
   
   ### Automated tests
   
   `filter with is not null on a struct field or an outer column`, added to 
[ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/cdf5292884895126825e4349938f2da3959fc812/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1058),
 compares Gluten with Spark on a struct-field predicate and an outer-column 
predicate.
   
   With the base commit's code, the new test fails:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-Dsuites="org.apache.gluten.functions.ScalarFunctionsValidateSuite filter"
   ```
   
   ```
   - map_filter
   - filter
   - filter with is not null on a struct field or an outer column *** FAILED ***
   …
     !== Correct Answer - 2 ==   == Gluten Answer - 2 ==
      struct<>                   struct<>
     ![ArrayBuffer([1])]         [ArrayBuffer([1], [null])]
      [ArrayBuffer([2])]         [ArrayBuffer([2])] (GlutenQueryTest.scala:460)
   …
   Tests: succeeded 2, failed 1, canceled 0, ignored 0, pending 0
   ```
   
   With this change:
   
   ```bash
   ./build/mvn -o -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite
   ```
   
   ```
   ScalarFunctionsValidateSuite:
   …
   - array_compact
   …
   - filter
   - filter with is not null on a struct field or an outer column
   …
   Tests: succeeded 91, failed 0, canceled 0, ignored 0, pending 0
   All tests passed.
   ```
   
   The Bolt backend change compiles:
   
   ```bash
   ./build/mvn -o -B compile -Pbackends-bolt -Pspark-3.5 -pl backends-bolt 
-Dspotless.check.skip=true -Dscalastyle.skip=true
   ```
   
   ```
   [INFO] BUILD SUCCESS
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to