Davis-Zhang-Onehouse opened a new pull request, #20705:
URL: https://github.com/apache/hudi/pull/20705
### Describe the issue this Pull Request addresses
Part 4 of 4 of the metadata table validator v2 series. **Stacked on #20704 →
#20703 → #20702; please review those first. The diff here includes all three.**
The commit this PR adds is the last one on the branch, "add
HoodieMetadataTableValidatorV2 behind --use-validator-v2".
#20702 explains the series and how v2 relates to
`HoodieMetadataTableValidator`. This PR wires the two dimensions from #20703
and #20704 into a runnable validator, opt-in from the existing tool.
### Summary and Changelog
Users get `--use-validator-v2` on `HoodieMetadataTableValidator`, default
`false`. With it set, latest file slice validation runs on v2; every other v1
validation stays as it is. Without it, v1 behaves exactly as before.
- `HoodieMetadataTableValidatorV2`:
- hosts the metadata table file system view once on the driver, behind an
embedded timeline server; executors read it through a remote view instead of
rebuilding it per partition;
- pins the metadata snapshot, reads the active timeline before and after
the run, and runs the duplicate base file and FS-MDT diff dimensions (each can
be switched off with `--validate-duplicate-base-files` /
`--validate-fs-mdt-diff`, but not both);
- fails only on positive proof;
- reports a run that cannot pin its snapshot, or that the archiver
overlapped, as *skipped* with a stable marker per cause, and never as a pass.
- `HoodieMetadataTableValidator`:
- adds `--use-validator-v2` and v2's options: `--validation-mode`
(fail-fast or exhaustive), `--validation-output-root` /
`--validation-output-dir` / `--validation-run-id` (where exhaustive mode writes
its reports), `--validate-duplicate-base-files`, `--validate-fs-mdt-diff`,
`--fs-view-storage-type`, `--fs-view-memory-fraction`,
`--fs-view-base-store-path`, `--partitions-per-task`, `--partition-threads`,
`--max-files-per-round`;
- moves v1's file-slice comparison helpers into
`HoodieMetadataValidationUtils` (added in #20704) unchanged, so both validators
share one implementation.
- `ValidationRunReports`, `MismatchReportWriter`, `DanglingReportWriter`,
`FileDiffReportWriter`: the per-run report files exhaustive mode writes, and a
run handle that records which comparisons were actually assessed. A run in
which the FS-MDT comparison did not run cannot be read as one that found
nothing.
- Two resource-leak fixes the driver-hosted view depends on:
- `SpillableMapBasedFileSystemView.closeResources()` closed none of its
maps. It called `super.closeResources()` first, which nulls the six map fields,
so every `close*IfPresent()` after it saw `null`. The spilled map file was left
for `deleteOnExit`, and under `ROCKS_DB` a whole RocksDB instance leaked per
view. The maps are now closed first, each guarded so one failure neither leaks
the others nor skips the superclass close; the first failure is rethrown.
Covered in `TestSpillableMapBasedFileSystemView`.
- `EmbeddedTimelineService.stopForBasePath()`: a failure releasing one
table's view (for example a metadata table close against object storage) no
longer skips stopping the shared server and its thread pool.
- End-to-end suites, run on local Spark against real tables: both
dimensions, exhaustive and fail-fast modes, archived-timeline adjudication,
reconciliation with pending cleans and rollbacks, base-path spellings, and
serialization of the state shipped to executors under bare Kryo.
### Impact
- New CLI options on `HoodieMetadataTableValidator`, all inert unless
`--use-validator-v2` is set. No existing option changes default or meaning.
- `SpillableMapBasedFileSystemView` now actually releases its spill files
and RocksDB instances on close. This affects every user of that view, not only
the validator.
- No storage format or public Java API change.
### Risk Level
low. The validator path is opt-in and off by default. The two shared-code
changes are leak fixes covered by tests: the close ordering is pinned in
`TestSpillableMapBasedFileSystemView`, and the timeline service change only
adds a catch around releasing one table's view. At this branch's head the
validator suites run 714 cases in `hudi-utilities` and 53 in
`TestSpillableMapBasedFileSystemView`, all green.
### Documentation Update
The new options carry their CLI descriptions. A follow-up can add a v2
section to the metadata table validator page on the website once the series
lands.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]