pedrumj2 opened a new pull request, #13622:
URL: https://github.com/apache/gluten/pull/13622

   ## What changes are proposed in this pull request?
   
   Spark fails 
[bitmap_construct_agg](https://spark.apache.org/docs/3.5.6/api/sql/index.html#bitmap_construct_agg)
 on a position outside the bitmap with 
[SparkArrayIndexOutOfBoundsException](https://github.com/apache/spark/blob/v3.5.5/common/utils/src/main/scala/org/apache/spark/SparkException.scala#L483)
 and error condition 
[INVALID_BITMAP_POSITION](https://github.com/apache/spark/blob/v3.5.5/common/utils/src/main/resources/error/error-classes.json#L1212-L1217).
 With this change, the Velox backend fails with the same exception instead of a 
[GlutenException](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-core/src/main/java/org/apache/gluten/exception/GlutenException.java).
   
   | Position | Before | After | Spark 3.5.5 without Gluten |
   |---|---|---|---|
   | 32768 | `GlutenException` wrapping Velox's `(32768 vs. 32768) Bitmap 
position out of bounds` | `SparkArrayIndexOutOfBoundsException: 
[INVALID_BITMAP_POSITION] The 0-indexed bitmap position 32768 is out of bounds. 
The bitmap has 32768 bits (4096 bytes).` | Same as after |
   | -1 | `GlutenException` wrapping Velox's `(-1 vs. 0) Bitmap position out of 
bounds` | `SparkArrayIndexOutOfBoundsException: [INVALID_BITMAP_POSITION] The 
0-indexed bitmap position -1 is out of bounds. The bitmap has 32768 bits (4096 
bytes).` | Same as after |
   
   Fixes apache/gluten#12238
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     export VELOX_BUILD_SHARED=ON
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write Spark's two test queries:
   
   ```bash
   cat > out_of_bounds.sql <<'EOF'
   SELECT bitmap_construct_agg(col) FROM VALUES (32768) AS tab(col);
   EOF
   cat > negative.sql <<'EOF'
   SELECT bitmap_construct_agg(col) FROM VALUES (-1) AS tab(col);
   EOF
   ```
   
   5. Run each query with Gluten, and print the exception it fails with and 
Velox's reason, if any:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     for q in out_of_bounds.sql negative.sql; do
       spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
         --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
         --conf spark.plugins=org.apache.gluten.GlutenPlugin \
         --conf spark.memory.offHeap.enabled=true \
         --conf spark.memory.offHeap.size=4g \
         --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
         -f "$q" 2>err.log
       grep -m 1 "^Job aborted" err.log | sed "s/.*executor driver): //"
       grep -m 1 "^Reason:" err.log
     done
   '
   ```
   
   On the base commit, both queries fail with a `GlutenException` that wraps 
Velox's error:
   
   ```
   org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Error during calling Java code 
from native code: org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
   Reason: (32768 vs. 32768) Bitmap position out of bounds
   org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Error during calling Java code 
from native code: org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
   Reason: (-1 vs. 0) Bitmap position out of bounds
   ```
   
   On this pull request's branch, rerunning steps 2 and 5 gives Spark's 
exception, with Velox's error kept as its cause:
   
   ```
   org.apache.spark.SparkArrayIndexOutOfBoundsException: 
[INVALID_BITMAP_POSITION] The 0-indexed bitmap position 32768 is out of bounds. 
The bitmap has 32768 bits (4096 bytes).
   Reason: (32768 vs. 32768) Bitmap position out of bounds
   org.apache.spark.SparkArrayIndexOutOfBoundsException: 
[INVALID_BITMAP_POSITION] The 0-indexed bitmap position -1 is out of bounds. 
The bitmap has 32768 bits (4096 bytes).
   Reason: (-1 vs. 0) Bitmap position out of bounds
   ```
   
   Spark without Gluten, which is step 5's `spark-sql` command keeping only 
`--master`, fails with the same exception:
   
   ```
   org.apache.spark.SparkArrayIndexOutOfBoundsException: 
[INVALID_BITMAP_POSITION] The 0-indexed bitmap position 32768 is out of bounds. 
The bitmap has 32768 bits (4096 bytes).
   org.apache.spark.SparkArrayIndexOutOfBoundsException: 
[INVALID_BITMAP_POSITION] The 0-indexed bitmap position -1 is out of bounds. 
The bitmap has 32768 bits (4096 bytes).
   ```
   
   ### Automated tests
   
   Inside the `apache/gluten:centos-9-jdk8` image, 
[GlutenQueryExecutionErrorsSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/errors/GlutenQueryExecutionErrorsSuite.scala)
 and 
[GlutenBitmapExpressionsQuerySuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenBitmapExpressionsQuerySuite.scala#L28)
 pass on Spark 3.5:
   
   ```bash
   ./build/mvn -ntp -B clean test -Pspark-3.5 -Pbackends-velox -Pspark-ut -pl 
gluten-ut/spark35 -am 
-DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -Dtest=none 
-DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.errors.GlutenQueryExecutionErrorsSuite,org.apache.spark.sql.GlutenBitmapExpressionsQuerySuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   GlutenQueryExecutionErrorsSuite:
   …
   - INVALID_BITMAP_POSITION: position out of bounds
   …
   - INVALID_BITMAP_POSITION: negative position
   …
   GlutenBitmapExpressionsQuerySuite:
   …
   - bitmap_construct_agg routes to native
   …
   Tests: succeeded 50, failed 0, canceled 0, ignored 3, pending 0
   All tests passed.
   ```
   
   On Spark 4.0, 4.1 and 4.2, the same suites pass with JDK 17 and 
`SPARK_ANSI_SQL_MODE=false`, which [Gluten's CI 
sets](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/.github/workflows/velox_backend_x86.yml#L65-L66):
   
   ```bash
   SPARK_ANSI_SQL_MODE=false ./build/mvn -ntp -B clean test -Pspark-4.0 
-Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -pl gluten-ut/spark40 -am 
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/" -Dtest=none 
-DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.errors.GlutenQueryExecutionErrorsSuite,org.apache.spark.sql.GlutenBitmapExpressionsQuerySuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   …
   - INVALID_BITMAP_POSITION: position out of bounds
   …
   - INVALID_BITMAP_POSITION: negative position
   …
   - bitmap_construct_agg routes to native
   …
   Tests: succeeded 64, failed 0, canceled 0, ignored 3, pending 0
   All tests passed.
   ```
   
   ```bash
   SPARK_ANSI_SQL_MODE=false ./build/mvn -ntp -B clean test -Pspark-4.1 
-Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -pl gluten-ut/spark41 -am 
-DargLine="-Dspark.test.home=/opt/shims/spark41/spark_home/" -Dtest=none 
-DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.errors.GlutenQueryExecutionErrorsSuite,org.apache.spark.sql.GlutenBitmapExpressionsQuerySuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   …
   - INVALID_BITMAP_POSITION: position out of bounds
   …
   - INVALID_BITMAP_POSITION: negative position
   …
   - bitmap_construct_agg routes to native
   …
   Tests: succeeded 68, failed 0, canceled 0, ignored 3, pending 0
   All tests passed.
   ```
   
   ```bash
   SPARK_ANSI_SQL_MODE=false ./build/mvn -ntp -B clean test -Pspark-4.2 
-Pscala-2.13 -Pjava-17 -Pbackends-velox -Pspark-ut -pl gluten-ut/spark42 -am 
-DargLine="-Dspark.test.home=/opt/shims/spark42/spark_home/" -Dtest=none 
-DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.errors.GlutenQueryExecutionErrorsSuite,org.apache.spark.sql.GlutenBitmapExpressionsQuerySuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   …
   - INVALID_BITMAP_POSITION: position out of bounds
   …
   - INVALID_BITMAP_POSITION: negative position
   …
   - bitmap_construct_agg routes to native
   …
   Tests: succeeded 68, failed 0, canceled 0, ignored 3, pending 0
   All tests passed.
   ```
   
   The change also compiles against Spark 3.4:
   
   ```bash
   ./build/mvn -ntp -B test-compile -Pbackends-velox -Pspark-ut -Pspark-3.4 -pl 
gluten-ut/spark34 -am
   ```
   
   ```
   [INFO] BUILD SUCCESS
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to