schumiyi opened a new pull request, #13657:
URL: https://github.com/apache/gluten/pull/13657

   ## What changes are proposed in this pull request?
   
   Fixes #13655.
   
   Velox emits POSEXPLODE ordinality as a one-based BIGINT, but the native 
Generate rewrite declares it using Spark's original INT attribute. When the 
normalization cast runs in `ColumnarPartialProjectExec` (for example, with 
`cast` blacklisted), JVM expression binding selects `getInt` on the BIGINT 
Arrow vector and fails before the cast executes.
   
   Use a separate `LongType` attribute with a fresh exprId for the native 
output and the cast input. The post projection continues to return Spark's 
zero-based INT position with its original exprId and qualifier. ARRAY/MAP and 
OUTER handling preserve the existing layout and marker semantics; a rejected 
native candidate restores the original Spark Generate.
   
   ## How was this patch tested?
   
   - Added `PullOutGenerateProjectSuite` under `gluten-ut/spark35`: 10 JVM 
tests cover schema/identity, ARRAY/MAP, inner/outer, null values and padding, 
rejected-candidate fallback, and Arrow/MutableProjection reading. A 
source-isolated harness using the exact production post-project method and 
actual rewrite manager was run with public Spark 3.5.5, Scala 2.12.18, Arrow 
15.0.0 and JDK 17: unpatched main failed 10/10; patched method passed 10/10. 
The unpatched Arrow case reproduced `UnsupportedOperationException` in 
`ArrowColumnVector.getInt`.
   - Added materialized Parquet integration coverage to 
`UDFPartialProjectSuite`: native Generate on/off, ARRAY/MAP, inner/outer, 
empty/null collections, and null elements/values. Assertions require native 
Generate and the partial project consuming its LONG ordinal, preventing 
complete fallback from producing a false pass.
   - `./dev/format-scala-code.sh`, `./.github/workflows/util/check.py header 
main --fix`, and `git diff --check` passed. The integration suite also passed 
Scala parsing.
   - Full Gluten/native build and native SQL integration were not run locally; 
the JVM harness does not replace those CI checks.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex (GPT-6.1-sol and GPT-6-astra)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to