schumiyi opened a new issue, #13655:
URL: https://github.com/apache/gluten/issues/13655

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   ## Backend and versions
   
   - Velox backend.
   - The affected rewrite is still present in Gluten `main` at 
`3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6`.
   - That revision selects `IBM/velox` branch `dft-2026_11_10`, which currently 
resolves to `67fa1bbee62e9f8752b3c34cc92f179022cd2ac6`.
   - The failure was originally observed in a downstream build. See the 
validation note below for the distinction between that observation, the 
current-main JVM regression, and a full native integration run.
   
   ## Problem
   
   Velox `UnnestNode` emits its ordinality column as **BIGINT**, with positions 
starting at **1**. Spark `posexplode` exposes an **INT** position starting at 
**0**.
   
   `PullOutGenerateProjectHelper.pullOutPostProject` accounts for the index 
difference with `CAST(position AS INT) - 1`, but currently also uses Spark's 
original **INT** attribute as the output attribute of the native Generate. The 
declared input type therefore disagrees with the native column's physical type.
   
   When that cast is evaluated on the JVM, the projection must first read the 
cast's input. Binding the input as INT selects `getInt` on a BIGINT Arrow 
vector, which throws `UnsupportedOperationException` before the conversion can 
execute. A cast around the incorrectly declared input cannot repair its reader.
   
   The original downstream stack included `ArrowWritableColumnVector.getInt` 
and `ColumnarPartialProjectExec`. Current `main` uses Spark Arrow column 
vectors instead of that removed Gluten vector implementation; the reader/type 
mismatch remains in the rewrite. The expected failure mechanism is the same, 
but those historical class names should not be mistaken for an observed 
current-main native stack.
   
   ## Reproduction
   
   In a Spark session configured with the Gluten Velox backend and its matching 
native libraries, use a materialized Parquet input so constant folding cannot 
bypass Generate:
   
   ```sql
   SET spark.gluten.enabled=false;
   
   CREATE TABLE posexplode_ordinality_repro (k INT, a ARRAY<INT>) USING PARQUET;
   
   INSERT INTO posexplode_ordinality_repro VALUES
     (0, array(10, 20)),
     (1, array(CAST(NULL AS INT), 30)),
     (2, array()),
     (3, CAST(NULL AS ARRAY<INT>));
   
   SET spark.gluten.enabled=true;
   SET spark.sql.ansi.enabled=false;
   SET spark.gluten.sql.columnar.generate=true;
   SET spark.gluten.sql.columnar.partial.project=true;
   SET spark.gluten.expression.blacklist=cast;
   
   SELECT k, pos, item
   FROM posexplode_ordinality_repro
   LATERAL VIEW posexplode(a) e AS pos, item
   ORDER BY k, pos;
   ```
   
   The relevant executed plan must contain a native `GenerateExecTransformer` 
whose ordinality is consumed by `ColumnarPartialProjectExec`; a plan that falls 
back the entire Generate does not exercise this path. Depending on the version 
and plan, the row conversion boundary can expose the same incorrect schema 
through another JVM reader.
   
   Also run the query with `LATERAL VIEW OUTER`, and the corresponding map 
input cases. The regression coverage should include real null elements/values, 
empty collections, and null collections.
   
   ## Expected behavior
   
   - The query matches vanilla Spark.
   - A real collection element retains its 0-based INT position, including an 
element/value that is itself null.
   - An OUTER padding row for an empty or null collection has a null position.
   - The externally visible position keeps Spark's original output attribute 
identity and schema.
   
   ## Root cause and proposed fix
   
   Create a separate native ordinality attribute using:
   
   ```scala
   val nativeOrdinal = originalOrdinal.withDataType(LongType).newInstance()
   ```
   
   Use it in the rewritten native Generate's output and as the input of the 
existing cast/subtraction. Keep the original Spark position's `exprId` and 
qualifier on the final alias.
   
   `LongType` makes JVM expression binding select `getLong`, matching the 
native BIGINT column. A fresh `exprId` distinguishes that internal 1-based LONG 
column from the externally visible 0-based INT result; the two attributes 
should not share one identity with different types and meanings. The type 
correction directly addresses the observed reader failure. Identity separation 
preserves the plan contract between those two representations.
   
   Apply this only to the rewritten native candidate. If native validation 
rejects the candidate, fallback must retain the original Spark Generate and its 
original INT output. The native ordinality layout and the OUTER marker behavior 
remain as defined by Velox. This is a Gluten schema-adaptation fix; changing 
Velox's BIGINT ordinality contract is unnecessary.
   
   Removing `cast` from the blacklist may avoid the JVM partial-project path 
for a particular query, but does not correct the declared native schema. Other 
JVM consumers/fallback paths can still encounter that mismatch.
   
   ## Source evidence
   
   - [Gluten's current position 
rewrite](https://github.com/apache/gluten/blob/3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6/backends-velox/src/main/scala/org/apache/gluten/execution/GenerateExecTransformer.scala#L271):
 uses `originalOrdinal` in both the native output and the cast.
   - [Velox output 
schema](https://github.com/IBM/velox/blob/67fa1bbee62e9f8752b3c34cc92f179022cd2ac6/velox/core/PlanNode.cpp#L1514):
 ordinality is BIGINT.
   - [Velox ordinality 
values](https://github.com/IBM/velox/blob/67fa1bbee62e9f8752b3c34cc92f179022cd2ac6/velox/exec/Unnest.cpp#L356):
 real rows use 1-based `int64_t`; OUTER padding uses zero before the 
marker-controlled projection.
   - [JVM partial 
projection](https://github.com/apache/gluten/blob/3671995dfbd2badbce6c209aa9d7f1ec6b0ac4e6/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala#L172):
 binds expressions to input attributes and evaluates them against Arrow batch 
rows.
   
   Related: #9315 added partial-project support for blacklisted built-in 
expressions. #12709 fixes a different OUTER `stack` marker/layout issue and 
does not correct POSEXPLODE's ordinality type.
   
   ## Validation
   
   A source-isolated JVM regression was run against the exact 
`pullOutPostProject` method from the above main revision and the proposed 
patch, using public Spark 3.5.5, Scala 2.12.18, Arrow 15.0.0, and JDK 17 
dependencies. The harness compiles the production method with the actual 
rewrite manager and runs the new `PullOutGenerateProjectSuite` without loading 
a native backend.
   
   - Unpatched main: 10 tests failed. The Arrow case reproduced 
`UnsupportedOperationException` from `ArrowColumnVector.getInt`; the schema and 
row-evaluation cases also exposed the incorrect INT binding.
   - Patched method: all 10 tests passed, including ARRAY/MAP, inner/outer, 
attribute identity, fallback restoration, and the Arrow/MutableProjection 
reader check.
   - This is not a full Gluten build or native SQL integration run. A 
Parquet-backed native regression is included in the proposed patch for CI; its 
plan assertions require both native Generate and the JVM partial project, so 
complete fallback cannot produce a false pass.
   
   AI assistance: Generated-by: OpenAI Codex (GPT-6.1-sol and GPT-6-astra).
   
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   spark.gluten.enabled=true
   spark.sql.ansi.enabled=false
   spark.gluten.sql.columnar.generate=true
   spark.gluten.sql.columnar.partial.project=true
   spark.gluten.expression.blacklist=cast
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   Source-isolated JVM regression against unpatched main (Spark 3.5.5):
   java.lang.UnsupportedOperationException
     at 
org.apache.spark.sql.vectorized.ArrowColumnVector$ArrowVectorAccessor.getInt(ArrowColumnVector.java:234)
     at 
org.apache.spark.sql.vectorized.ArrowColumnVector.getInt(ArrowColumnVector.java:85)
     at 
org.apache.spark.sql.vectorized.ColumnarBatchRow.getInt(ColumnarBatchRow.java:106)
   
   Patched method: Tests: succeeded 10, failed 0.
   Full native SQL integration has not been run locally.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to