ravjot28 opened a new pull request, #20587:
URL: https://github.com/apache/hudi/pull/20587
### Describe the issue this Pull Request addresses
Closes #20577
The RocksDB file system view applies an incremental sync by adding or
removing single files in its stored slices. After
a compaction is scheduled, log files can end up in another slice than a
listing assigns them to, or in two slices, and
a rolled-back delta commit can drop the pending compaction slice. The
in-memory and spillable views rebuild their file
groups and are correct.
### Summary and Changelog
The incrementally synced RocksDB view equals a view built by listing, in
every order of events covered below.
- `RocksDbBasedFileSystemView#applyDeltaFileSlicesToPartitionView`: for each
changed file group, read its stored
slices, add or remove the delta's base and log files, build the group with
the code a listing uses, and replace the
group's stored slices with the result. The private per-slice merge helpers
it replaces are removed; the public
`fetchLatestFileSliceBeforeOrOn` stays as it is.
- New tests in `TestIncrementalFSViewSync` (run for the in-memory, spillable
and RocksDB views through its subclasses):
`testCompactionScheduledWhileListedDeltaCommitIsInflight`,
`testOneSyncSeesTheCompletedDeltaCommitAndTheScheduledCompaction`,
`testOneSyncSeesTheListedDeltaCommitCompletedAndTheScheduledCompaction`,
`testRollbackOfADeltaCommitKeepsThePendingCompactionSlice`,
`testCompactionScheduledAfterRollbackLeavingLogFile`.
Each compares the synced view with a freshly built one.
### Impact
No API or configuration change. Only the `EMBEDDED_KV_STORE` view with
incremental timeline sync changes. A sync now
reads and rewrites every stored slice of each changed file group instead of
single slices; the cost is per changed
file group, not per partition.
#20579 removes complete rebuilds that repair this state today; please merge
this fix first or together with it.
**Backport:** please include this fix in the next 1.2.x patch release; the
defect is present in 1.2.1. The commit does
not cherry-pick cleanly onto `release-1.2.1`: `RocksDbBasedFileSystemView`
conflicts with a variable rename made on
master after 1.2.1 (`df` to `baseFile`). I can open an adapted change
against `branch-1.2.x`.
### Risk Level
low. The file group is built by the same code a listing uses; views other
than the RocksDB view are unchanged.
Tests, run on master with JDK 11 and passing checkstyle checks for main and
test sources (scalastyle did not execute
in this Java-only reactor), with `mvn test -pl hudi-hadoop-common -am
-Dtest=...` and the classes listed below:
- Without the main code change: the five new cases fail for the RocksDB view
only
(`TestRocksDBBasedIncrementalFSViewSync` 5 of 10), each on a comparison
with a freshly built view; the in-memory and
spillable runs of the same tests pass (10/10 each).
- With the change: `TestIncrementalFSViewSync` 10/10,
`TestRocksDBBasedIncrementalFSViewSync` 10/10,
`TestSpillableMapBasedIncrementalFSViewSync` 10/10,
`TestRocksDbBasedFileSystemView` 52/52 pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]