Davis-Zhang-Onehouse opened a new pull request, #20702:
URL: https://github.com/apache/hudi/pull/20702
### Describe the issue this Pull Request addresses
`HoodieMetadataTableValidator` compares the metadata table (MDT) with a
storage listing, but on large tables its latest-file-slice check does not scale
and cannot tell a real inconsistency from a transient one. It builds both file
system views per partition on the executors, resolves the archived timeline
once per file group, and reports any difference without saying whether the
timeline explains it. This series adds **`HoodieMetadataTableValidatorV2`**, a
separate, opt-in validator for the latest file slices. It is reached from the
existing tool with `--use-validator-v2` (off by default), so nothing changes
for current callers.
How v2 relates to the existing validator:
- v1 stays as it is and remains the default. Every v1 validation (column
stats, bloom filters, record index, partitions, and so on) stays v1-only.
- v2 covers latest file slice validation only, as two independent dimensions:
- **duplicate base files**: a file slice holding more than one base file;
- **FS-MDT diff**: files that storage and the MDT disagree on.
- v2 adjudicates every difference against the timeline (active and archived,
plus pending clean, rollback and compaction state). A run fails only on
positive proof. A difference it cannot settle is reported as not settled, and
never as clean.
- v2 hosts the MDT file system view once on the driver, behind an embedded
timeline server, so executors no longer rebuild it per partition.
The series is stacked; each PR compiles and passes its own tests at its
position:
1. **This PR**: the archived timeline batch resolver and the shared
primitives (library code only, nothing calls it yet).
2. The duplicate base file dimension.
3. The FS-MDT diff dimension with timeline adjudication.
4. The `HoodieMetadataTableValidatorV2` entry point, run reports, the
`--use-validator-v2` flag on `HoodieMetadataTableValidator`, two small
resource-leak fixes it depends on
(`SpillableMapBasedFileSystemView.closeResources`, `EmbeddedTimelineService`
shutdown), and the end-to-end suites.
PRs 2–4 are opened as drafts stacked on this one. Please review this one
first.
### Summary and Changelog
Adds library code under `org.apache.hudi.utilities.util` (and
`util.archive`), with unit tests. No existing class is modified.
- `util.archive.*`: resolves a **batch** of instants against the archived
timeline in one scan per instant range, for both the V1 (log-format commits
archive) and V2 (LSM timeline) layouts. The validator previously read the
archive once per file group. `ArchiveInstantSearcher` locates the archive files
that can hold a range, `HoodieArchiveFileScanner` reads them and reports
damaged or partially merged files as such, and `ArchivedInstantWorkingSet`
holds what was found.
- `InstantState`, `InstantStateIndex`, `InstantBuckets`,
`InstantRangePartitioner`, `InstantRangeBalancer`: a serializable index of
every active instant's state, read once on the driver so executors cannot
disagree about which instants completed, and a total range partitioner (a
binary search over sorted boundaries) that gives each Spark task a contiguous
instant range without splitting an instant.
- `CommitWriteStats`, `CommitPathProjector`, `ArchivedCommitPaths`,
`PartitionPaths`: what a commit recorded for a file group. "The commit named
nothing", "named several" and "could not be read" stay three different answers.
- `MismatchRecord`, `MismatchPattern`, `MismatchSide`: the finding model
shared by both dimensions.
- `Inconclusive`, `InconclusiveCause`, `ValidationFailures`,
`ValidationMode`: the outcome vocabulary. A run that could not decide says why,
with a stable marker per cause, and is never reported as a pass.
- `Branch`, `BranchBeacon`: an enumerated decision-branch surface, logged at
debug and used by tests to assert which branch decided a verdict.
- `ArchivalProbe`, `RecomputeProbe`, `ExecutorMetaClientFactory`,
`CloseableExecutor`, `JavaSerializedPayload`: helpers for detecting that the
archiver overlapped a run, rebuilding meta clients on executors, and
broadcasting driver state independently of the configured Spark serializer.
No code was copied from outside the project.
### Impact
None at runtime: no existing code path reaches the new classes until PR 4
wires them behind `--use-validator-v2`. There are no public API, storage format
or config changes.
### Risk Level
none. This is new library code with no callers in this PR. Its unit tests
(155 cases across 20 classes) run in `hudi-utilities`.
### Documentation Update
none in this PR. The `--use-validator-v2` flag and the v2-specific options
arrive in PR 4, with their CLI descriptions.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]