iemejia opened a new pull request, #12612:
URL: https://github.com/apache/gluten/pull/12612
## What changes are proposed in this pull request?
(Note: `apache/gluten` has no `master` branch, so this PR targets the
default branch `main`.)
This is a follow-up to #12390 that makes native Delta deletion-vector (DV)
materialization both correct and cheaper by sourcing the Delta table root from
the authoritative place instead of inferring it.
### Background
Delta DV descriptors reference the on-disk bitmap using a
table-root-relative UUID path. To read those bitmaps during planning,
`DeltaDeletionVectorScanInfo.normalize` needs the Delta table root. Previously
the root was inferred from each data file path via a heuristic
`resolveTablePath`:
1. take the data file's parent directory,
2. walk up one level per partition column,
3. probe the filesystem for a `_delta_log` directory,
4. if that failed, keep walking parents probing `_delta_log` at each level,
5. and custom-unescape `%`-encoded path characters along the way.
This is fragile (it depends on file layout and partition depth, and can
resolve the wrong root for non-trivial paths) and it performs at least one
`FileSystem.exists("_delta_log")` call per `normalize` invocation.
### This change
- `DeltaScanTransformer.getSplitInfosFromPartitions` now reads the table
root directly from `relation.location` when it is a `TahoeFileIndex` (which
also covers `PreparedDeltaFileIndex` and the other Tahoe subclasses used for
time travel, path-based reads, DML, and CDC). `TahoeFileIndex.path` is the
authoritative Delta table root.
- That root is threaded into
`DeltaDeletionVectorScanInfo.normalize(partitionFiles, tablePath)` and
`extract(...)` across all Delta profiles (2.3 / 2.4 / 3.3 / 4.0).
- The heuristic `resolveTablePath`, `isDeltaTablePath` (the `_delta_log`
filesystem probe), and the manual `unescapePathName` helper are removed.
- Non-Tahoe, format-only scans keep the generic split representation
unchanged. Delta does not attach per-file DV metadata to such scans, so this is
safe.
- The `DeltaPlanningBenchmark` is updated to the explicit table-root API.
### Why this is an improvement
- **Correctness:** the table root now comes from Delta itself rather than
being guessed from partition nesting and filesystem probing. This removes a
class of latent mis-resolution bugs for partitioned tables and tables at
non-standard paths.
- **Performance (planning):** one filesystem existence probe per `normalize`
call is eliminated. On local storage this is a small, consistent win; on remote
object stores (S3/ABFS/HDFS) an `exists()` is a network round-trip per DV
split, so the saving is proportionally larger there — consistent with the
remote-storage motivation of the parent tracking issue #12399.
### Test hardening
The DV unit tests were strengthened so they actually prove the supplied root
is used: the synthetic `PartitionedFile` now points at an unrelated directory
while the real table root is supplied separately, and the tests require a
table-root-relative UUID (`storageType == "u"`) DV. The native
partitioned-table integration test now asserts that the DELETE really produced
an on-disk DV before validating query results, guarding the removal of the old
partition-count walk-up.
## How was this patch tested?
- Scala formatting via `./dev/format-scala-code.sh` (JDK 17).
- Compilation across Delta 2.3 (compat), Velox Delta 3.3 / Spark 3.5, and
Velox Delta 4.0 / Spark 4.0, including backend test sources.
- `DeltaDeletionVectorScanInfoSuite` on Delta 3.3 and Delta 4.0: 4/4 tests
pass on each.
- `git diff --check` clean.
- `DeltaPlanningBenchmark` (`normalize`, 100 DV files x 10k rows, 200 timed
iters, alternating fresh JVMs) comparing current `main` vs `main` + this
change: median 61.40 µs/file vs 61.91 µs/file (~0.8% faster, ~51 µs saved per
100-file call) on local storage, with every run of the change faster than every
baseline run. Larger gains are expected on remote filesystems where the removed
`exists()` probe is a network round-trip.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenCode github-copilot/gpt-5.6-sol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
