This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch 
gh-readonly-queue/master/pr-20622-f740680aa40862bf008f0f9c6ada11cb8d35e37c
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 948706b059e41b24e3918e6362dba38f788dd0f8
Author: Danny Chan <[email protected]>
AuthorDate: Wed Nov 18 03:53:41 2026 +0000

    docs(spark): document SDP truncate and full-refresh limitations (#20622)
---
 hudi-integ-test/spark-pipelines/ACCEPTANCE.md      | 46 ++++++++++++++++++++++
 .../spark-pipelines/MATERIALIZED_VIEWS.md          | 11 ++++--
 2 files changed, 53 insertions(+), 4 deletions(-)

diff --git a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md 
b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
index 8357c1e07e19..cfcb14196414 100644
--- a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
+++ b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
@@ -67,6 +67,52 @@ fan-in, chaining, or commit/checkpoint failure injection.
 The generated source, SQL and pipeline specification are retained in the logs 
as
 runnable examples within the harness's isolated filesystem layout.
 
+## Truncate and full-refresh limitations
+
+The release scope agreed in 
[#20404](https://github.com/apache/hudi/issues/20404)
+retains the existing full-table truncate behavior. Timeline-preserving truncate
+and streaming checkpoint-generation isolation are deferred together; they are
+not prerequisites for this release's SDP integration.
+
+| Operation | Current behavior |
+| --- | --- |
+| `TRUNCATE TABLE t` | Deletes the table directory and re-initializes the 
table from its properties. Data, timeline history, savepoints, metadata table, 
and stored sink batch IDs are lost. |
+| `TRUNCATE TABLE t PARTITION (...)` | Uses a `delete_partition` replace 
commit; it does not purge and re-initialize the whole table. |
+| SDP full refresh of a streaming table | Purges the target and restarts the 
flow from a fresh SDP-managed checkpoint. Old sink batch IDs are removed with 
the table, so they cannot suppress replay starting at batch 0. |
+| Ordinary streaming restart | Reuses the existing checkpoint and sink 
identifier to skip batches already committed to Hudi. |
+
+Full-table truncate is non-transactional. Do not run it while another writer is
+active. Deletion and table re-initialization can fail independently; a failed
+truncate does not guarantee that the previous snapshot survives. SDP's
+truncate–alter–append sequence is also non-atomic: a failure after truncation
+can leave an empty table, and a failure during replay can leave only the 
batches
+committed so far. A successful refresh with empty input leaves an empty table.
+Savepoints and retention settings cannot preserve history across this physical
+purge, and the old metadata table is not retained.
+
+Incremental/streaming reader offsets into the old table history become invalid.
+Downstream consumers are **not reset automatically** when their source is
+truncated or recreated. Stop affected consumers before the reset, then refresh
+or rebuild downstream tables and their checkpoints in dependency order along
+with the upstream table. This applies to chained SDP flows and to 
HoodieStreamer
+using `HoodieIncrSource`. Do not assume a downstream reader will detect the 
reset
+and fail safely; default failure on source recreation is follow-up work.
+
+Keep the checkpoint and sink identifier on ordinary restarts. Deleting a
+checkpoint in place while retaining the target's sink metadata is unsupported:
+batch IDs restart at zero and can be skipped against an old high-water mark.
+Checkpoint relocation requires an explicit rebuild/migration; changing a path
+alone does not isolate a new generation in the current sink. SDP full refresh
+coordinates the target purge with a fresh checkpoint. Replay requires the 
source
+data needed to rebuild the target to remain available.
+
+These are operational limitations, not additional acceptance-test results.
+Timeline-preserving truncate (the proposal in
+[#20378](https://github.com/apache/hudi/pull/20378)) must ship together with
+checkpoint-generation isolation and an explicit legacy-identifier migration
+contract if pursued after this release. Atomic refresh and automatic downstream
+recovery are not provided by the current implementation.
+
 ## Incremental Hudi table chaining
 
 `chaining_acceptance.py` adds a second partitioned Hudi streaming table that
diff --git a/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md 
b/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
index 03d783c99d38..2fd3f2d0ee8b 100644
--- a/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
+++ b/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
@@ -39,10 +39,13 @@ Keyless aggregate materialized views remain deferred under 
#20405.
 Land this scenario after the #20453 harness dependency. Checks remain active 
under
 `python -O`, and SIGTERM unwinds the harness for server and 
temporary-directory cleanup.
 
-This is Phase 2 coverage on today's purge-style truncate. History retention,
-atomic refresh and timeline-preserving truncate are intentionally outside this
-scenario and remain Phase 4 work. Metadata indexing is disabled. The test does
-not establish streaming-table chaining or full-refresh behavior.
+This is Phase 2 coverage on today's purge-style truncate. See the
+[truncate and full-refresh 
limitations](ACCEPTANCE.md#truncate-and-full-refresh-limitations)
+for history loss, the non-atomic refresh window, and downstream rebuild 
requirements.
+Timeline-preserving truncate and checkpoint-generation isolation are deferred
+together beyond this release under #20404; atomic refresh is also out of scope.
+Metadata indexing is disabled. The test does not establish streaming-table
+chaining or streaming-table full-refresh behavior.
 
 The harness bounds each client/CLI process and removes its temporary data.
 Logs and generated pipeline definitions remain under 
`target/mv-acceptance-logs`.

Reply via email to