ravjot28 opened a new pull request, #20589:
URL: https://github.com/apache/hudi/pull/20589

   ### Describe the issue this Pull Request addresses
   
   Closes #20579
   
   `TimelineDiffHelper` lists as new every instant of the new timeline that is 
not equal to one of the old timeline, and
   instant equality includes the state. A pending compaction that moves from 
requested to inflight, and a pending log
   compaction on every sync (the old side of the comparison holds no log 
compactions), are added again by the incremental
   sync; the duplicate file group check throws and the view is rebuilt from 
scratch, dropping every loaded partition.
   
   ### Summary and Changelog
   
   A file system view with incremental timeline sync stays incremental while 
compactions and log compactions are pending.
   
   - `TimelineDiffHelper#getNewInstantsForIncrementalSync`: leaves out of the 
new instants every pending compaction or log
     compaction that was already pending in the old timeline and is still 
pending (its transition is computed from the
     existing pending transitions). Finished, removed and new pending instants 
keep their handling; the duplicate file
     group check stays strict for a different instant.
   - New test class `TestIncrementalFSViewSyncTransitions` (in-memory, 
spillable and RocksDB views):
     `testCompactionStateTransitionsSyncIncrementally`, 
`testLogCompactionStateTransitionSyncsIncrementally`,
     `testPendingLogCompactionStaysOutOfLaterSyncs`, 
`testCompactionAndLogCompactionPendingTogetherSyncIncrementally`,
     `testViewCreatedWhileALogCompactionIsPendingRebuildsOnce`, 
`testUnschedulingAPendingCompactionRebuildsTheView`,
     `testDifferentCompactionInstantForAPendingFileGroupIsRefused`. The tests 
check initialization counts, pending
     operations, completion, unscheduling and duplicate rejection; the 
compaction transition, the unrelated delta commit
     and the combined compaction and log compaction cases also compare against 
a fresh view.
     `testViewCreatedWhileALogCompactionIsPendingRebuildsOnce` covers a view 
created while a log compaction is already
     pending: its first sync still rebuilds once, and later syncs are 
incremental (without this change every later sync
     rebuilds as well). The last two pin the behaviour that does not change: an 
unscheduled compaction still rebuilds, a
     different instant for a pending file group is still refused.
   - The combined case runs a compaction and a log compaction pending at the 
same time on two file groups of one
     partition: both move to inflight, a delta commit lands on the log 
compacted file group, the compaction completes,
     and the view stays incremental and equal to a fresh one. The older
     `TestIncrementalFSViewSync#testAsyncMajorAndMinorCompaction` is disabled 
on master for an unrelated reason and is not
     enabled here.
   
   ### Impact
   
   No API or configuration change. Only views with 
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default
   `false`) change: they keep their loaded partitions when a compaction moves 
to inflight or while a log compaction is
   pending.
   
   A view created while a log compaction is already pending still rebuilds 
once, on its first sync: the view manager
   builds views with `filterCompletedAndCompactionInstants()`, which holds no 
log compactions, so the old side of the
   first diff does not know the instant. Later syncs are incremental.
   
   The rebuilds removed here currently repair two other defects of 
incrementally synced views, fixed in separate pull
   requests: the RocksDB view can place a log file in another file slice than a 
listing does
   (#20587), and file groups of untouched partitions keep the timeline of their 
last rebuild
   (#20588). Please merge #20587 before this one or
   together with it: the rebuild removed here currently repairs its incorrect 
log file placement. Merging
   #20588 first is recommended as well, so that this change does not extend the
   retention of earlier timelines; that one affects memory only.
   
   **Backport:** please include this fix in the next 1.2.x patch release. The 
commit cherry-picks cleanly onto
   `release-1.2.1`.
   
   ### Risk Level
   
   medium until the prerequisite view fixes are merged. This change removes 
rebuilds that currently repair incorrect
   RocksDB log file placement and release retained earlier timelines. After 
those fixes land, it only suppresses
   re-adding pending operations already held by the view.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (scalastyle did not execute
   in this Java-only reactor), with `mvn test -pl hudi-hadoop-common -am 
-Dtest=...` and the classes listed below:
   - Without the main code change: 15 of the 21 cases of 
`TestIncrementalFSViewSyncTransitions` fail on initialization
     counts (five methods, three view types): 
`testCompactionStateTransitionsSyncIncrementally`,
     `testLogCompactionStateTransitionSyncsIncrementally`, 
`testPendingLogCompactionStaysOutOfLaterSyncs`,
     `testCompactionAndLogCompactionPendingTogetherSyncIncrementally` (expected 
1, was 2) and
     `testViewCreatedWhileALogCompactionIsPendingRebuildsOnce` (expected 2, was 
3).
   - With the change: `TestIncrementalFSViewSyncTransitions` 21/21, 
`TestIncrementalFSViewSync` 5/5,
     `TestRocksDBBasedIncrementalFSViewSync` 5/5, 
`TestSpillableMapBasedIncrementalFSViewSync` 5/5 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to