ravjot28 opened a new pull request, #20588:
URL: https://github.com/apache/hudi/pull/20588

   ### Describe the issue this Pull Request addresses
   
   Closes #20578
   
   `HoodieFileGroup` keeps the timeline it was built with. An incremental sync 
rebuilds only the partitions the synced
   instants touched, so every other loaded partition kept the timeline of its 
last rebuild, and a long-lived view such as
   the timeline server's held one timeline object per loaded partition. The 
spillable view's serializer kept the same old
   timelines in a map for spilled partitions.
   
   ### Summary and Changelog
   
   After an incremental sync every stored file group refers to the view's 
current timeline, and the view holds one
   timeline object.
   
   - `HoodieFileGroup`: a copy constructor that binds the copy to another 
timeline.
   - `IncrementalTimelineSyncFileSystemView` / `HoodieTableFileSystemView`: 
after an incremental sync the view replaces,
     under its write lock, every stored file group bound to another timeline 
with a copy bound to the view's timeline.
     The view's timeline here is the one complete builds use 
(`getVisibleCommitsAndCompactionTimeline()`); until now an
     incremental sync bound each rebuilt partition to its own 
`filterCompletedAndCompactionInstants()` copy. A copy, not
     a setter, so that file groups already handed to readers keep the timeline 
they were read with.
   - `SpillableMapBasedFileSystemView` rebinds the file groups it holds in 
memory (`ExternalSpillableMap#inMemoryValues`)
     and never reads its disk entries for this. It builds its 
`HoodieFileGroupSerializer` with the new constructor that
     takes a supplier of the view's current timeline: every group read back is 
bound to it and the serializer keeps no
     timeline map (a `null` supplier is refused); a read back after the view 
was closed fails with a message saying so.
     The existing no-argument constructor keeps its behaviour.
   - `AbstractTableFileSystemView` (and `getAllFileGroups()` of the in-memory 
and spillable views): the reads that
     traverse every stored file group (`getLatestBaseFiles()`,
     `getLatestBaseFilesInRange`, `getLatestFileSliceInRange`, 
`getAllFileGroups()`) are collected while the read lock is
     held. A spilled file group is now bound to the view's timeline when it is 
read back, so it must be read back under
     the lock; a lazy stream traversed after the lock was released could also 
mix the state before and after a sync. The
     per-partition reads already returned copies and are unchanged.
   - The RocksDB view needs no file group rebinding. It shares the collected 
all-partition reads of the base class. File
     groups already returned to callers keep their timeline.
   - Tests: new classes `TestIncrementalFSViewSyncTimelineRebinding` and 
`TestIncrementalFSViewSyncSnapshotReads`;
     `TestHoodieFileGroupSerializer` gains three tests: two for the supplier 
and one that refuses a null supplier.
   
   ### Impact
   
   No API removed: `HoodieFileGroupSerializer` gains a constructor that takes a 
`Supplier<HoodieTimeline>`, and
   `HoodieFileGroup` a copy constructor; `ExternalSpillableMap#inMemoryValues` 
is added. Memory: a view keeps one timeline
   object instead of one per loaded partition. Cost: every incremental sync 
copies all in-memory file groups (one
   `TreeMap` and one `HoodieFileGroup` per stored group, under the write lock), 
also a sync with no new instant, because
   the view installs a new timeline object on every refresh; for 100,000 file 
groups with three slices each this took 6
   to 11 ms warm (about 100 ms cold).
   
   The four all-partition reads above now build their result list under the 
read lock for every view type, also with
   incremental sync disabled. This raises peak memory for large results; the 
spillable view's `getAllFileGroups()` can
   materialize the groups of every spilled partition. Their callers in the code 
base collect the stream anyway. The
   rebinding timings above do not measure this cost.
   
   This fix can land independently. Merging it before #20579 is recommended:
   that change removes rebuilds that currently release these earlier timelines.
   
   **Backport:** please include this fix in the next 1.2.x patch release; the 
defect is present in 1.2.1. The commit
   cherry-picks cleanly onto `release-1.2.1`.
   
   ### Risk Level
   
   medium. The change touches the stored state of the in-memory and spillable 
views on every incremental sync and the
   read path of the all-partition reads. Mitigation: the new tests compare the 
synced views with fresh ones and check a
   lazy all-partition read traversed during a sync; the existing file system 
view, incremental sync, spillable map and
   remote view suites pass.
   
   Tests, run on master with JDK 11 and passing checkstyle checks for main and 
test sources (scalastyle did not execute
   in this Java-only reactor), with `mvn test -pl hudi-timeline-service -am 
-Dtest=...` and the classes listed below:
   - Without the main code change (the master serializer test, the rebinding 
test view built with the no-argument
     serializer): `TestIncrementalFSViewSyncTimelineRebinding` 3 of 3 fail on 
the timeline identity (expected one
     timeline object, was 2; a spilled group read back is not bound to the 
current timeline) and
     `TestIncrementalFSViewSyncSnapshotReads` 2 of 2 on a traversal that shows 
two states.
   - With the change: `TestHoodieFileGroupSerializer` 4/4, 
`TestHoodieFileGroup` 9/9, `TestHoodieTableFileSystemView` 51/51,
     `TestSpillableMapBasedFileSystemView` 51/51, `TestIncrementalFSViewSync` 
5/5,
     `TestSpillableMapBasedIncrementalFSViewSync` 5/5, 
`TestIncrementalFSViewSyncTimelineRebinding` 3/3,
     `TestIncrementalFSViewSyncSnapshotReads` 2/2, `TestExternalSpillableMap` 
32/32, `TestPriorityBasedFileSystemView` 31/31,
     `TestRemoteHoodieTableFileSystemView` 54/54, `TestRequestHandler` 2/2 pass.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to