ravjot28 opened a new issue, #20578:
URL: https://github.com/apache/hudi/issues/20578

   ### Bug Description
   
   **What happened:**
   
   With incremental timeline sync, an incremental sync rebuilds only the 
partitions the synced instants touched and then
   refreshes the view's own timeline. The file groups of every other loaded 
partition keep a reference to the timeline of
   their last rebuild. After many syncs that each touch a few partitions, a 
long-lived view (above all the timeline
   server's) holds one `HoodieTimeline` object per loaded partition, each with 
its instant list and derived maps. On a
   table with a long active timeline and thousands of loaded partitions these 
take a large part of the heap.
   
   `SpillableMapBasedFileSystemView` has the same problem for the partitions it 
holds in memory. For spilled partitions,
   `HoodieFileGroupSerializer` keeps a map from file group id to the timeline 
the group was last written with and binds
   the group it reads back to that timeline, so spilled partitions keep their 
old timelines alive too and come back bound
   to them.
   
   A related effect: a read over every partition returned as a lazy stream and 
traversed while a sync runs can mix the
   state before and after the sync.
   
   **What you expected:**
   
   After a sync every stored file group refers to the view's current timeline.
   
   **Steps to reproduce:**
   
   1. Create a table with partitions `p1` and `p2`, each with completed file 
groups.
   2. Build a `HoodieTableFileSystemView` (or 
`SpillableMapBasedFileSystemView`) with
      `hoodie.filesystem.view.incr.timeline.sync.enable=true` and load both 
partitions.
   3. Complete a commit that writes only to `p1` and call `view.sync()`.
   4. Collect the timeline objects of `view.getAllFileGroups()` by identity: 
two objects; the groups of `p2` still refer
      to the timeline from before the sync.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-common/src/main/java/org/apache/hudi/common/model/HoodieFileGroup.java:71,76,93`:
 a file group keeps the
     timeline it was built with and its last instant.
   - 
`hudi-common/src/main/java/org/apache/hudi/common/table/view/IncrementalTimelineSyncFileSystemView.java:92-106`:
     `maySyncIncrementally` applies the new instants, which rebuilds the file 
groups of the touched partitions only
     (`applyDeltaFileSlicesToPartitionView`, `:406-453`), then refreshes only 
the view's own timeline (`:106`).
   - `IncrementalTimelineSyncFileSystemView.java:277,383,450-452`: the rebuilt 
partitions are bound to
     `timeline.filterCompletedAndCompactionInstants()`, a new timeline object 
per partition and instant, so touched
     partitions do not share the view's timeline object either.
   - 
`hudi-common/src/main/java/org/apache/hudi/common/serialization/HoodieFileGroupSerializer.java:42,55,65`:
 the
     serializer keeps a file group id to timeline map and binds read-back 
groups to it; the spillable view creates it
     without access to the view's timeline
     
(`hudi-common/src/main/java/org/apache/hudi/common/table/view/SpillableMapBasedFileSystemView.java:100`).
   
   Related: #12110 added the serializer's timeline map; #15497 proposes turning 
incremental timeline sync on by default.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
   **Query engine:** any; most visible on the timeline server (embedded or 
standalone)
   **Relevant configs:** 
`hoodie.filesystem.view.incr.timeline.sync.enable=true` (default `false`); 
in-memory or spillable
   view
   
   ### Logs and Stack Trace
   
   none (memory retention)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to