pedrumj2 opened a new pull request, #13639:
URL: https://github.com/apache/gluten/pull/13639

   ## What changes are proposed in this pull request?
   
   Spark fails an insert of a char or varchar value that is longer than its 
declared length with 
[EXCEED_LIMIT_LENGTH](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java#L31),
 while the Velox backend failed it with a 
[GlutenException](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-core/src/main/java/org/apache/gluten/exception/GlutenException.java)
 carrying Velox's own message 
([CharVarcharUtils.cpp](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/CharVarcharUtils.cpp#L37-L38)).
 With this change, the Velox backend fails with Spark's exception.
   
   | Inserted value | Spark 3.5.5 without Gluten | Gluten before this change | 
Gluten after this change |
   |---|---|---|---|
   | `'12345'` into `VARCHAR(5)` | Inserted | Inserted | Inserted |
   | `'123456'` into `VARCHAR(5)` | `EXCEED_LIMIT_LENGTH` | `GlutenException` | 
`EXCEED_LIMIT_LENGTH` |
   | `'123456'` into `CHAR(5)` | `EXCEED_LIMIT_LENGTH` | `GlutenException` | 
`EXCEED_LIMIT_LENGTH` |
   
   Part of apache/gluten#11912
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     source /opt/rh/gcc-toolset-12/enable
     export VELOX_BUILD_SHARED=ON
     cd /work/gluten
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write three queries that insert from a Parquet table:
   
   ```bash
   cat > fits.sql <<'EOF'
   DROP TABLE IF EXISTS src;
   DROP TABLE IF EXISTS t;
   CREATE TABLE src USING parquet AS SELECT '12345' AS s;
   CREATE TABLE t (c VARCHAR(5)) USING parquet;
   INSERT INTO t SELECT s FROM src;
   SELECT c FROM t;
   EOF
   cat > varchar.sql <<'EOF'
   DROP TABLE IF EXISTS src;
   DROP TABLE IF EXISTS t;
   CREATE TABLE src USING parquet AS SELECT '123456' AS s;
   CREATE TABLE t (c VARCHAR(5)) USING parquet;
   INSERT INTO t SELECT s FROM src;
   EOF
   cat > char.sql <<'EOF'
   DROP TABLE IF EXISTS src;
   DROP TABLE IF EXISTS t;
   CREATE TABLE src USING parquet AS SELECT '123456' AS s;
   CREATE TABLE t (c CHAR(5)) USING parquet;
   INSERT INTO t SELECT s FROM src;
   EOF
   ```
   
   5. Run each query with Gluten, and print the exception it fails with and 
Velox's reason, if any:
   
   ```bash
   docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c 
'
     cd /work/gluten
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     for q in fits.sql varchar.sql char.sql; do
       spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
         --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
         --conf spark.plugins=org.apache.gluten.GlutenPlugin \
         --conf spark.memory.offHeap.enabled=true \
         --conf spark.memory.offHeap.size=4g \
         --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
         -f "$q" 2>err.log
       grep -m 1 "Job aborted" err.log | sed "s/.*executor driver): //"
       grep -m 1 "^Reason:" err.log
     done
   '
   ```
   
   On the base commit, both too-long values fail with a `GlutenException` that 
wraps Velox's error:
   
   ```
   12345
   org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
   Reason: (6 vs. 5) Exceeds allowed length limitation: 5
   org.apache.gluten.exception.GlutenException: 
org.apache.gluten.exception.GlutenException: Exception: VeloxUserError
   Reason: (6 vs. 5) Exceeds allowed length limitation: 5
   ```
   
   On this pull request's branch, rerunning steps 2 and 5 gives Spark's 
exception, with Velox's error kept as its cause:
   
   ```
   12345
   org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds 
char/varchar type length limitation: 5.
   Reason: (6 vs. 5) Exceeds allowed length limitation: 5
   org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds 
char/varchar type length limitation: 5.
   Reason: (6 vs. 5) Exceeds allowed length limitation: 5
   ```
   
   Spark without Gluten, which is step 5's `spark-sql` command keeping only 
`--master`, wraps the same exception in `TASK_WRITE_FAILED`:
   
   ```
   12345
   org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while 
writing rows to file:/work/gluten/spark-warehouse/t.
   org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while 
writing rows to file:/work/gluten/spark-warehouse/t.
   ```
   
   ```bash
   grep -m 1 "^Caused by: org.apache.spark.SparkRuntimeException" err.log
   ```
   
   ```
   Caused by: org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] 
Exceeds char/varchar type length limitation: 5.
   ```
   
   ### Automated tests
   
   Inside the `apache/gluten:centos-9-jdk8` image, the char/varchar suites pass 
on Spark 3.5:
   
   ```bash
   ./build/mvn -B clean install -Pbackends-velox -Pspark-3.5 -Pspark-ut 
-DskipTests -Dspotless.check.skip=true -Dscalastyle.skip=true 
-Dcheckstyle.skip=true
   ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl 
gluten-ut/spark35 -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" 
-Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.GlutenFileSourceCharVarcharTestSuite,org.apache.spark.sql.GlutenDSV2CharVarcharTestSuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   GlutenFileSourceCharVarcharTestSuite:
   …
   - length check for input string values: nested in map key
   …
   - Gluten - length check for input string values: partitioned columns
   - Gluten - char/varchar type values length check: partitioned columns of 
other types
   …
   Tests: succeeded 101, failed 0, canceled 0, ignored 2, pending 0
   All tests passed.
   ```
   
   On Spark 3.4, with the same two commands using `-Pspark-3.4`, 
`gluten-ut/spark34` and `/opt/shims/spark34/spark_home/`:
   
   ```
   GlutenFileSourceCharVarcharTestSuite:
   …
   - length check for input string values: nested in map key
   …
   - Gluten - length check for input string values: partitioned columns
   - Gluten - char/varchar type values length check: partitioned columns of 
other types
   …
   Tests: succeeded 100, failed 0, canceled 0, ignored 2, pending 0
   All tests passed.
   ```
   
   On Spark 4.0, 4.1 and 4.2, with JDK 17 and `SPARK_ANSI_SQL_MODE=false`, 
which [Gluten's CI 
sets](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/.github/workflows/velox_backend_x86.yml#L65-L66):
   
   ```bash
   ./build/mvn -B clean install -Pbackends-velox -Pspark-4.0 -Pscala-2.13 
-Pjava-17 -Pspark-ut -DskipTests -Dspotless.check.skip=true 
-Dscalastyle.skip=true -Dcheckstyle.skip=true
   SPARK_ANSI_SQL_MODE=false ./build/mvn -B test -Pbackends-velox -Pspark-4.0 
-Pscala-2.13 -Pjava-17 -Pspark-ut -pl gluten-ut/spark40 
-DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/" -Dtest=none 
-DfailIfNoTests=false 
-DwildcardSuites=org.apache.spark.sql.GlutenFileSourceCharVarcharTestSuite,org.apache.spark.sql.GlutenDSV2CharVarcharTestSuite
 -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true
   ```
   
   ```
   GlutenFileSourceCharVarcharTestSuite:
   …
   - length check for input string values: nested in map key
   …
   Tests: succeeded 118, failed 0, canceled 1, ignored 0, pending 0
   All tests passed.
   ```
   
   Spark 4.1 runs the same commands with `-Pspark-4.1`, `gluten-ut/spark41` and 
`/opt/shims/spark41/spark_home/`:
   
   ```
   Tests: succeeded 118, failed 0, canceled 1, ignored 0, pending 0
   All tests passed.
   ```
   
   Spark 4.2 runs them with `-Pspark-4.2`, `gluten-ut/spark42` and no 
`-DargLine`, adding 
[GlutenResolveDefaultColumnsSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenResolveDefaultColumnsSuite.scala)
 to `-DwildcardSuites`:
   
   ```
   GlutenResolveDefaultColumnsSuite:
   …
   - SPARK-57187: non-foldable default exceeding CHAR/VARCHAR length fails at 
INSERT time (explicit DEFAULT keyword)
   …
   Tests: succeeded 138, failed 0, canceled 1, ignored 0, pending 0
   All tests passed.
   ```
   
   The canceled test on Spark 4.0, 4.1 and 4.2 is [SPARK-48792: Fix INSERT with 
partial column list to a table with 
char/varchar](https://github.com/apache/spark/blob/v4.0.2/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala#L687),
 which Spark cancels itself with an `assume`.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to