pedrumj2 opened a new pull request, #13639: URL: https://github.com/apache/gluten/pull/13639
## What changes are proposed in this pull request? Spark fails an insert of a char or varchar value that is longer than its declared length with [EXCEED_LIMIT_LENGTH](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java#L31), while the Velox backend failed it with a [GlutenException](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-core/src/main/java/org/apache/gluten/exception/GlutenException.java) carrying Velox's own message ([CharVarcharUtils.cpp](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/functions/sparksql/CharVarcharUtils.cpp#L37-L38)). With this change, the Velox backend fails with Spark's exception. | Inserted value | Spark 3.5.5 without Gluten | Gluten before this change | Gluten after this change | |---|---|---|---| | `'12345'` into `VARCHAR(5)` | Inserted | Inserted | Inserted | | `'123456'` into `VARCHAR(5)` | `EXCEED_LIMIT_LENGTH` | `GlutenException` | `EXCEED_LIMIT_LENGTH` | | `'123456'` into `CHAR(5)` | `EXCEED_LIMIT_LENGTH` | `GlutenException` | `EXCEED_LIMIT_LENGTH` | Part of apache/gluten#11912 ## How was this patch tested? ### Reproduction 1. Clone the repo and check out the base commit: ```bash git clone https://github.com/apache/gluten.git cd gluten git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746 ``` 2. Build the native libraries and the bundle jar: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable export VELOX_BUILD_SHARED=ON cd /work/gluten ./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests ' ``` 3. Fetch Spark 3.5.5: ```bash curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz ``` 4. Write three queries that insert from a Parquet table: ```bash cat > fits.sql <<'EOF' DROP TABLE IF EXISTS src; DROP TABLE IF EXISTS t; CREATE TABLE src USING parquet AS SELECT '12345' AS s; CREATE TABLE t (c VARCHAR(5)) USING parquet; INSERT INTO t SELECT s FROM src; SELECT c FROM t; EOF cat > varchar.sql <<'EOF' DROP TABLE IF EXISTS src; DROP TABLE IF EXISTS t; CREATE TABLE src USING parquet AS SELECT '123456' AS s; CREATE TABLE t (c VARCHAR(5)) USING parquet; INSERT INTO t SELECT s FROM src; EOF cat > char.sql <<'EOF' DROP TABLE IF EXISTS src; DROP TABLE IF EXISTS t; CREATE TABLE src USING parquet AS SELECT '123456' AS s; CREATE TABLE t (c CHAR(5)) USING parquet; INSERT INTO t SELECT s FROM src; EOF ``` 5. Run each query with Gluten, and print the exception it fails with and Velox's reason, if any: ```bash docker run --rm -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' cd /work/gluten BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) for q in fits.sql varchar.sql char.sql; do spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ -f "$q" 2>err.log grep -m 1 "Job aborted" err.log | sed "s/.*executor driver): //" grep -m 1 "^Reason:" err.log done ' ``` On the base commit, both too-long values fail with a `GlutenException` that wraps Velox's error: ``` 12345 org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError Reason: (6 vs. 5) Exceeds allowed length limitation: 5 org.apache.gluten.exception.GlutenException: org.apache.gluten.exception.GlutenException: Exception: VeloxUserError Reason: (6 vs. 5) Exceeds allowed length limitation: 5 ``` On this pull request's branch, rerunning steps 2 and 5 gives Spark's exception, with Velox's error kept as its cause: ``` 12345 org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5. Reason: (6 vs. 5) Exceeds allowed length limitation: 5 org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5. Reason: (6 vs. 5) Exceeds allowed length limitation: 5 ``` Spark without Gluten, which is step 5's `spark-sql` command keeping only `--master`, wraps the same exception in `TASK_WRITE_FAILED`: ``` 12345 org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/work/gluten/spark-warehouse/t. org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/work/gluten/spark-warehouse/t. ``` ```bash grep -m 1 "^Caused by: org.apache.spark.SparkRuntimeException" err.log ``` ``` Caused by: org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5. ``` ### Automated tests Inside the `apache/gluten:centos-9-jdk8` image, the char/varchar suites pass on Spark 3.5: ```bash ./build/mvn -B clean install -Pbackends-velox -Pspark-3.5 -Pspark-ut -DskipTests -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -Pspark-ut -pl gluten-ut/spark35 -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.spark.sql.GlutenFileSourceCharVarcharTestSuite,org.apache.spark.sql.GlutenDSV2CharVarcharTestSuite -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ``` ``` GlutenFileSourceCharVarcharTestSuite: … - length check for input string values: nested in map key … - Gluten - length check for input string values: partitioned columns - Gluten - char/varchar type values length check: partitioned columns of other types … Tests: succeeded 101, failed 0, canceled 0, ignored 2, pending 0 All tests passed. ``` On Spark 3.4, with the same two commands using `-Pspark-3.4`, `gluten-ut/spark34` and `/opt/shims/spark34/spark_home/`: ``` GlutenFileSourceCharVarcharTestSuite: … - length check for input string values: nested in map key … - Gluten - length check for input string values: partitioned columns - Gluten - char/varchar type values length check: partitioned columns of other types … Tests: succeeded 100, failed 0, canceled 0, ignored 2, pending 0 All tests passed. ``` On Spark 4.0, 4.1 and 4.2, with JDK 17 and `SPARK_ANSI_SQL_MODE=false`, which [Gluten's CI sets](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/.github/workflows/velox_backend_x86.yml#L65-L66): ```bash ./build/mvn -B clean install -Pbackends-velox -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pspark-ut -DskipTests -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true SPARK_ANSI_SQL_MODE=false ./build/mvn -B test -Pbackends-velox -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pspark-ut -pl gluten-ut/spark40 -DargLine="-Dspark.test.home=/opt/shims/spark40/spark_home/" -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.spark.sql.GlutenFileSourceCharVarcharTestSuite,org.apache.spark.sql.GlutenDSV2CharVarcharTestSuite -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true ``` ``` GlutenFileSourceCharVarcharTestSuite: … - length check for input string values: nested in map key … Tests: succeeded 118, failed 0, canceled 1, ignored 0, pending 0 All tests passed. ``` Spark 4.1 runs the same commands with `-Pspark-4.1`, `gluten-ut/spark41` and `/opt/shims/spark41/spark_home/`: ``` Tests: succeeded 118, failed 0, canceled 1, ignored 0, pending 0 All tests passed. ``` Spark 4.2 runs them with `-Pspark-4.2`, `gluten-ut/spark42` and no `-DargLine`, adding [GlutenResolveDefaultColumnsSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenResolveDefaultColumnsSuite.scala) to `-DwildcardSuites`: ``` GlutenResolveDefaultColumnsSuite: … - SPARK-57187: non-foldable default exceeding CHAR/VARCHAR length fails at INSERT time (explicit DEFAULT keyword) … Tests: succeeded 138, failed 0, canceled 1, ignored 0, pending 0 All tests passed. ``` The canceled test on Spark 4.0, 4.1 and 4.2 is [SPARK-48792: Fix INSERT with partial column list to a table with char/varchar](https://github.com/apache/spark/blob/v4.0.2/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala#L687), which Spark cancels itself with an `assume`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
