Davis-Zhang-Onehouse opened a new pull request, #20705:
URL: https://github.com/apache/hudi/pull/20705

   ### Describe the issue this Pull Request addresses
   
   Part 4 of 4 of the metadata table validator v2 series. **Stacked on #20704 → 
#20703 → #20702; please review those first. The diff here includes all three.** 
The commit this PR adds is the last one on the branch, "add 
HoodieMetadataTableValidatorV2 behind --use-validator-v2".
   
   #20702 explains the series and how v2 relates to 
`HoodieMetadataTableValidator`. This PR wires the two dimensions from #20703 
and #20704 into a runnable validator, opt-in from the existing tool.
   
   ### Summary and Changelog
   
   Users get `--use-validator-v2` on `HoodieMetadataTableValidator`, default 
`false`. With it set, latest file slice validation runs on v2; every other v1 
validation stays as it is. Without it, v1 behaves exactly as before.
   
   - `HoodieMetadataTableValidatorV2`:
     - hosts the metadata table file system view once on the driver, behind an 
embedded timeline server; executors read it through a remote view instead of 
rebuilding it per partition;
     - pins the metadata snapshot, reads the active timeline before and after 
the run, and runs the duplicate base file and FS-MDT diff dimensions (each can 
be switched off with `--validate-duplicate-base-files` / 
`--validate-fs-mdt-diff`, but not both);
     - fails only on positive proof;
     - reports a run that cannot pin its snapshot, or that the archiver 
overlapped, as *skipped* with a stable marker per cause, and never as a pass.
   - `HoodieMetadataTableValidator`:
     - adds `--use-validator-v2` and v2's options: `--validation-mode` 
(fail-fast or exhaustive), `--validation-output-root` / 
`--validation-output-dir` / `--validation-run-id` (where exhaustive mode writes 
its reports), `--validate-duplicate-base-files`, `--validate-fs-mdt-diff`, 
`--fs-view-storage-type`, `--fs-view-memory-fraction`, 
`--fs-view-base-store-path`, `--partitions-per-task`, `--partition-threads`, 
`--max-files-per-round`;
     - moves v1's file-slice comparison helpers into 
`HoodieMetadataValidationUtils` (added in #20704) unchanged, so both validators 
share one implementation.
   - `ValidationRunReports`, `MismatchReportWriter`, `DanglingReportWriter`, 
`FileDiffReportWriter`: the per-run report files exhaustive mode writes, and a 
run handle that records which comparisons were actually assessed. A run in 
which the FS-MDT comparison did not run cannot be read as one that found 
nothing.
   - Two resource-leak fixes the driver-hosted view depends on:
     - `SpillableMapBasedFileSystemView.closeResources()` closed none of its 
maps. It called `super.closeResources()` first, which nulls the six map fields, 
so every `close*IfPresent()` after it saw `null`. The spilled map file was left 
for `deleteOnExit`, and under `ROCKS_DB` a whole RocksDB instance leaked per 
view. The maps are now closed first, each guarded so one failure neither leaks 
the others nor skips the superclass close; the first failure is rethrown. 
Covered in `TestSpillableMapBasedFileSystemView`.
     - `EmbeddedTimelineService.stopForBasePath()`: a failure releasing one 
table's view (for example a metadata table close against object storage) no 
longer skips stopping the shared server and its thread pool.
   - End-to-end suites, run on local Spark against real tables: both 
dimensions, exhaustive and fail-fast modes, archived-timeline adjudication, 
reconciliation with pending cleans and rollbacks, base-path spellings, and 
serialization of the state shipped to executors under bare Kryo.
   
   ### Impact
   
   - New CLI options on `HoodieMetadataTableValidator`, all inert unless 
`--use-validator-v2` is set. No existing option changes default or meaning.
   - `SpillableMapBasedFileSystemView` now actually releases its spill files 
and RocksDB instances on close. This affects every user of that view, not only 
the validator.
   - No storage format or public Java API change.
   
   ### Risk Level
   
   low. The validator path is opt-in and off by default. The two shared-code 
changes are leak fixes covered by tests: the close ordering is pinned in 
`TestSpillableMapBasedFileSystemView`, and the timeline service change only 
adds a catch around releasing one table's view. At this branch's head the 
validator suites run 714 cases in `hudi-utilities` and 53 in 
`TestSpillableMapBasedFileSystemView`, all green.
   
   ### Documentation Update
   
   The new options carry their CLI descriptions. A follow-up can add a v2 
section to the metadata table validator page on the website once the series 
lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to