Davis-Zhang-Onehouse opened a new pull request, #20703:
URL: https://github.com/apache/hudi/pull/20703

   ### Describe the issue this Pull Request addresses
   
   Part 2 of 4 of the metadata table validator v2 series. **Stacked on #20702; 
please review that first. The diff here includes it.** The commit this PR adds 
is the last one on the branch, "add the duplicate base file dimension for 
metadata table validator v2".
   
   #20702 explains the series and how v2 relates to 
`HoodieMetadataTableValidator`.
   
   A retried Spark write task can leave a second base file for the same file 
slice (same file group, same base instant, different write token), and only one 
of them is recorded by the commit. The file system view collapses the set to 
one member, so ordinary latest-file-slice validation never sees the extra file, 
and it stays on storage unnoticed. This PR adds v2's first validation 
dimension, which finds those slices and says which member is unaccounted for.
   
   ### Summary and Changelog
   
   Library code under `org.apache.hudi.utilities.util`, with unit tests. 
Nothing calls it until the last PR of the series.
   
   - `DuplicateBaseFileDetector`: phase 1, per partition. It reads the raw 
storage listing only (no metadata view, no timeline) and reports every file 
slice holding two or more base files. It parses names with 
`FSUtils`/`HoodieBaseFile`, so it cannot disagree with the rest of Hudi about 
what a file is called. Log files are not examined.
   - `DuplicateInstantWindow`: tolerates a duplicate only if its writing 
instant could have been in flight between two active-timeline reads that 
bracket the run. A retried task racing the validation is therefore not a 
finding, but a duplicate under an instant present at neither read is.
   - `DanglingAdjudicator`: decides, from the completed timeline and the 
archived instants resolved by #20702's batch resolver, whether a commit still 
names the file. The rule is membership, not exactly-one, because MOR write 
stats also name log files. Anything short of proof is `NOT_SETTLED`, and 
nothing throws.
   - `DuplicateBaseFileDimension`: runs on the executors. Fail-fast mode raises 
at the classify stage. Exhaustive mode keys the survivors by writing instant 
and range-partitions them, so each task scans one contiguous archived range 
once, then adjudicates.
   
   ### Impact
   
   None at runtime until the last PR wires the dimension behind 
`--use-validator-v2` (default off). No public API, storage format or config 
change.
   
   ### Risk Level
   
   none. New library code with no callers. Unit tests cover 56 cases across 
`TestDuplicateBaseFileDetector`, `TestDuplicateInstantWindow`, 
`TestDanglingAdjudicator` and `TestDuplicateBaseFileDimension`. The end-to-end 
suites for this dimension arrive with the entry point in the last PR.
   
   ### Documentation Update
   
   none in this PR. `--validate-duplicate-base-files` arrives in the last PR, 
with its CLI description.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(utilities): add th... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to