malinjawi opened a new pull request, #12836: URL: https://github.com/apache/gluten/pull/12836
## What changes are proposed in this pull request? > [!NOTE] > This draft is stacked on #12612. The first three commits are the authoritative `TahoeFileIndex` table-root work from that PR; the final two commits are this change. The stack should be rebased onto `main` after #12612 is merged. Move on-disk Delta deletion-vector payload reads off the Spark driver and add an opt-in path that reads the stored DV range directly through Velox. The current native scan eagerly opens every DV sidecar, reads and checksum-validates the bitmap, and embeds the payload bytes into the split before Spark submits tasks. On a 2,461-file S3 table this added approximately 100 seconds to each physical scan while the executors were idle. This patch introduces two compatible execution paths: - The default deferred path carries a serializable DV source in the Spark input partition and materializes it on an executor. Successful task-local materialization is memoized; failed reads remain retryable. - The opt-in native path carries the JVM-resolved absolute DV URI, stored-entry offset, payload size, and cardinality. Velox reads the exact range through `FileHandleFactory` and `BufferedInputBuilder`, allowing existing split preloading, I/O concurrency, and `AsyncDataCache` to apply. The native reader validates the four-byte stored length, payload bounds, CRC32, roaring bitmap encoding, and expected cardinality before applying row filtering. Inline DVs continue to use the existing JVM-materialized byte field. The original protobuf field is retained for compatibility and rollback. The native path is gated by: ```properties spark.gluten.sql.columnar.delta.deletionVector.nativeRangeRead.enabled=true ``` It is disabled by default and takes precedence over `deferPayloadRead.enabled`. Setting it back to `false` returns to executor-JVM materialization; setting `spark.gluten.sql.columnar.filescan=false` remains the broader rollback. The patch also adds driver/task DV metrics and native runtime counters for descriptor preparation, read attempts, bytes, and elapsed read time. ## How was this patch tested? - `./dev/format-scala-code.sh` - `./dev/format-cpp-code.sh` with clang-format 15.0.7 - `./dev/gen-all-config-docs.sh` - `git diff --check` - Compiled the Delta modules and modified test sources for Spark 3.3, 3.4, 3.5, 4.0, and 4.1 profiles. - Built and linked the Velox backend and `velox_delta_read_test` target. - `DeltaDeletionVectorScanInfoSuite`: 7 tests passed, including concurrent deferred materialization, failed-read retry, authoritative native descriptor handoff, and zero JVM payload reads. - `GlutenRuntimeConfigSuite`: 5 tests passed. - Added native tests for stored-range length, CRC32, bitmap/cardinality validation, and descriptor propagation. - Added Spark/Delta handoff tests for native filtering with zero JVM DV payload reads. The native test executable cannot run on the development macOS host because an unrelated Folly F14 assertion aborts during static initialization before GoogleTest begins. The target compiles and links successfully; Linux CI and the S3 benchmark are required before enabling the opt-in native path by default. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM BOB -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
