yihua opened a new pull request, #5234:
URL: https://github.com/apache/hudi/pull/5234
## What is the purpose of the pull request
A Hudi file slice may contain uncommitted log files, e.g., from
`AbstractTableFileSystemView::getLatestFileSlices` API. When the file system
view is based on the file system, uncommitted log files are included for
constructing Hudi file slices. When the file system view is based on metadata
table, uncommitted log files are excluded. Such behavior difference does not
cause any correctness issue as the compaction and clustering planning using
`getLatestFileSlices()` skips the uncommitted log files when doing log block
reading and merging. However, such behavior causes the metadata table
validator to fail.
To fix the problem, this PR adds the logic to exclude uncommitted log files
when comparing two file slices with the same base file, if there is no equality.
## Brief change log
- Adds logic to exclude uncommitted log files when checking equality of two
file slices in `HoodieMetadataTableValidator`
## Verify this pull request
A Hudi MOR table is constructed where there is an inflight deltacommit with
written but uncommitted log files. Without this PR,
`HoodieMetadataTableValidator` fails with `--validate-latest-file-slices`
because of uncommitted log files. After this PR,
`HoodieMetadataTableValidator` succeeds with warning messages about uncommitted
log files.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
