malinjawi opened a new pull request, #12269: URL: https://github.com/apache/gluten/pull/12269
What changes are proposed in this pull request? Split from:https://github.com/apache/gluten/pull/12198 This PR is the next split in the Delta deletion-vector scan stack after #12197, which has now landed. It adds the JVM/Substrait/Velox handoff that consumes the essential Delta DV scan info extracted by #12197, materializes serialized DV payloads on the JVM side, and passes them to native scan execution. Main changes: - add a Delta DV preprocessing rule for the Velox Delta component without replacing Delta's `PrepareDeltaScan` - reuse `DeltaDeletionVectorScanInfo` from #12197 to extract per-file DV metadata and serialized DV bytes from Delta-prepared scan files - add Delta local files Substrait nodes/builders carrying `DeltaReadOptions` - embed the serialized DV payload in `DeltaReadOptions`, instead of passing essential DV data through generic metadata columns - add a native `DeltaSplitInfo` path for Delta-specific split metadata - wire the handoff through `VeloxIteratorApi`, `VeloxPlanConverter`, `WholeStageResultIterator`, and `SubstraitToVeloxPlan` - strip Spark's synthetic DV predicate/internal columns only after the native split has the payload, so Velox applies the DV filter natively and avoids double filtering - add Spark 3.5 and Spark 4.0 focused handoff coverage This PR is intentionally handoff-only: - #12197 reviewed and landed the DV scan info extraction utility - performance/benchmark iteration remains a follow-up after the correctness handoff shape is reviewed - DELETE/UPDATE/MERGE DV DML support remains in later split work Issue: #11901 Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM BOB -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
