malinjawi opened a new pull request, #11963:
URL: https://github.com/apache/gluten/pull/11963
## What changes are proposed in this pull request?
This PR takes the direction we settled on after the earlier native DV PoC
work and the follow-up benchmarks.
The earlier PoC showed that a fully native Delta DV read path is feasible.
But the benchmark results did not show a strong performance reason to keep
native DV IO/materialization as the default design. The hotter path is still
native DV consume/apply/filter during scan.
So this PR changes the read path to:
- JVM / Delta side materializes the deletion vector and serializes it into
bytes
- Java passes the serialized DV payload to C++
- native / Velox keeps scan-time responsibilities:
- consume/deserialization of the serialized DV payload
- DV application and filtering during scan
- Delta split / connector / datasource integration
In other words, this PR keeps the native execution-side work from the
earlier PoC, but moves DV materialization/loading to the JVM side for a simpler
and easier-to-maintain baseline.
Additional cleanup in this PR:
- remove native DV file/inline/materialization ownership from this path
- strip the synthetic Spark DV predicate/column from the native offloaded
plan so DV is not applied twice
- add native Delta tests for the retained consume/apply path
So the intended split is now:
- JVM / Delta utilities own DV materialization/loading
- native / Velox owns scan-time DV consume/apply/filter
This also keeps the door open to add a native DV IO layer later if future
DV-heavy workloads justify it.
## How was this patch tested?
Build / validation:
- `./build/mvn -pl gluten-delta,backends-velox -am -Pbackends-velox
-Pspark-3.5 -Pdelta -DskipTests compile`
Functional validation:
- Real uncached Spark 3.5 Delta DV benchmark on a `4,000,000` row table with
`64` DV-bearing files
- Verified Gluten offloads through `FileDeltaScanTransformer`
- Verified result correctness matches vanilla Spark exactly
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM BOB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
