nsivabalan commented on issue #20404:
URL: https://github.com/apache/hudi/issues/20404#issuecomment-6644867515

   @danny0405 a design question for the checkpoint-generation part of this 
issue, before #20378 is picked back up.
   
   #20378 currently scopes the sink identifier to the checkpoint location 
(`default_single_writer@<checkpointLocation>`). An alternative is to keep the 
identifier stable and record the checkpoint location next to the batch id. 
That's the same idea as HoodieStreamer's checkpoint reset key:
   
   - **What HoodieStreamer does.** `--checkpoint X` is applied only if `X` 
differs from the reset key recorded in the last commit 
(`StreamerCheckpointUtils`). Once applied, the new commit records `X` as the 
reset key, so later restarts don't re-apply it. Resetting is an explicit act: 
you change the key.
   - **The sink analogue.** Commit metadata stores `{identifier: {batchId: N, 
checkpointLocation: <loc>}}`, or a separate reset-key entry. On `addBatch`:
     - same location: today's dedup (`batchId <= N` → skip);
     - different location: this is a new checkpoint generation (SDP full 
refresh, or a moved or recreated checkpoint dir), and batch ids are not 
comparable. Either treat it as a reset (don't skip; record the new location on 
the first commit), or fail unless the user acknowledges the reset with a 
config, matching the streamer's explicit-reset model. For SDP, full refresh 
would provide that acknowledgement automatically.
   
   **Why this might be preferable:**
   - One identifier per writer. Fan-in dedup (now tracked separately) keys on 
the flow, not on flow × location.
   - A location change can be detected and reported, instead of silently 
becoming a new key that leaves the old key's entries behind in the timeline.
   - It gives operators the same reset mental model as HoodieStreamer.
   
   **Open points:**
   - Is checkpoint-location equality a reliable generation signal? For example, 
could SDP reuse the same path across a full refresh?
   - Upgrade from today's bare `{identifier: batchId}` entries.
   
   Do you see any issue with taking this approach in #20378, versus the current 
location-scoped identifier?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to