nsivabalan commented on issue #20404:
URL: https://github.com/apache/hudi/issues/20404#issuecomment-6762486498

   @danny0405 @yihua proposing to descope this from the SDP release.
   
   SDP is new in this release, so we'd like to keep truncate as it is today 
(delete the table directory and re-initialize it) and document what that means, 
rather than change `TRUNCATE TABLE` semantics now. Today's truncate is already 
safe for SDP full refresh: it removes the sink's stored batch id along with the 
table, so a refresh starting from a fresh checkpoint at batch 0 doesn't skip 
data. Preserving the timeline is what would make the checkpoint-generation 
change necessary, so both parts move out together.
   
   **Documented behaviour for this release:**
   - `TRUNCATE TABLE t` (full table) deletes the table and re-initializes it. 
Timeline history, savepoints, the metadata table and all incremental/streaming 
reader offsets are lost. It is not transactional, and running it while another 
writer is active is unsupported.
   - `TRUNCATE TABLE t PARTITION (...)` is unchanged: it is a 
`delete_partition` replacecommit.
   - SDP full refresh of a streaming table: supported. The table is purged and 
the flow restarts from a fresh checkpoint.
   - Downstream tables that read a truncated or recreated table incrementally 
(chained flows, or HoodieStreamer with HoodieIncrSource) aren't reset 
automatically. Refresh them along with the upstream table. The same applies 
today when a HoodieStreamer source table is deleted and restarted.
   
   **Follow-ups after this release, if the community is interested:**
   - Truncate as a replacecommit that keeps history (#20378, to be parked), 
together with checkpoint-generation isolation in the sink.
   - Downstream readers fail by default when their source is recreated or 
truncated (HoodieIncrSource and the Spark streaming source).
   
   If there are no objections, I'll update the Phase 4 entry on #20401, park 
#20378, and add the limitation to the SDP docs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to