nsivabalan commented on issue #20401:
URL: https://github.com/apache/hudi/issues/20401#issuecomment-6373044659
Proposed plan: deliver RFC-112 in three phases, starting with an MVP
To get something working end to end sooner, I'd like to split the work
into three phases:
1. MVP: a single Hudi streaming table fed from an external source.
2. Materialized views, chaining, and keyed deletes, built on today's
truncate.
3. Timeline-preserving truncate and full refresh.
Truncate goes last because today's truncate is a purge. It also clears the
sink's batch-id metadata, so leaving it as is doesn't make full refresh lose
data. What has to wait is changing truncate to keep the timeline: that
change and the checkpoint-scoped sink id must ship together, and they're the
whole of Phase 3.
Phase 1: MVP (single streaming table, incremental upsert/insert)
Scope: a Kafka (or similar) → Hudi streaming table run repeatedly through
spark-pipelines, with each run picking up only new data. A PoC on Spark 4.1.1
Connect has already run this flow successfully with a small catalog shim.
Unsupported for now: --full-refresh, materialized views, truncate.
1. #20412 → #20413 → #20415 (#20400, sink timeline server and write-client
lifetime). Ready, and review feedback has been addressed. @danny0405, feel
free to take over, fix any minor issues, and land them. These go first
because steps 4–5 change the same sink code.
2. #20423 (#20369, saveAsTable append onto an existing table). I'll review
it today. The MVP doesn't need it, but Phase 2 does.
3. #20371, only the part the MVP needs:
- override createTable(Column[]) so tables created by SDP get
hoodie.properties
- make SetProperty/RemoveProperty idempotent in alterTable (SDP
re-sends every property on each run, so today the second run fails)
4. #20370: throw from the sink instead of calling System.exit(1), which
kills the shared Connect server.
5. #20403: pass per-flow spark.hoodie.* settings to the streaming sink
(needed to choose insert instead of the default upsert).
6. #20402: Spark Connect + spark-pipelines harness and CI lane.
Steps 3–6 can be developed in parallel.
7. MVP acceptance: end-to-end tests and a runnable example on the harness.
Two or more runs, exact row contents checked after each run, a restart with
no new data produces no new commit, and both upsert dedup and insert
are covered. This takes the incremental part of #20406 and the upsert part of
#20408.
Phase 2: materialized views, streaming-table chaining, keyed deletes (on
today's truncate)
Builds on Phase 1. The three tracks below can run in parallel.
Track A: materialized views
- A1. Rest of #20371:
- Treat DeleteColumn of _hoodie_* columns as a no-op, or hide those
columns from Table.columns(). SDP asks to delete them on every MV run after the
first.
- Raise a clear UnsupportedOperationException instead of MatchError for
the other ColumnChanges.
- A2. #20405: MV flows and repeated refresh. Depends on #20423 (append
onto an existing table) and A1. In this phase each refresh uses today's
truncate,
so the MV is fully recomputed and doesn't keep history across refreshes.
The timeline assertions in #20405 move to Phase 3.
Track B: Hudi → Hudi streaming-table chaining
- B1. #20407: the downstream streaming table reads the upstream table
incrementally.
- Define the starting offsets and how a restart resumes.
- Fail with a clear "rebuild required" error when history has expired or
the upstream table was physically recreated. Physical recreation matters in
this phase, because that is what today's truncate does on an upstream
full refresh.
- Document the AvailableNow backlog limitation (the Hudi source has no
admission control yet).
Track C: keyed upsert/delete and streaming-table hardening
- C1. Rest of #20408:
- _hoodie_is_deleted delete markers, out-of-order ordering values,
deletes followed by reinsertion
- negative tests for missing or invalid key, partition and precombine
fields
Depends on #20403.
- C2. Rest of #20406:
- AvailableNow completion, stop and failure all release sink resources
- reject unsupported fan-in (several flows writing one target) before
any write
- failure and restart tests between the Hudi commit and the Spark
checkpoint
- C3. Follow-up from #20400: the metadata-table writer still starts and
stops its own embedded timeline server on every commit (to be filed).
Phase 3: truncate and full refresh
Each item depends on 3.1.
- 3.1. #20404 (draft #20378): full-table TRUNCATE becomes one
replacecommit (an empty insert_overwrite_table), so the timeline, savepoints
and metadata
table are kept. The old purge stays behind an opt-in config. It ships
with a sink checkpoint identifier scoped to each checkpoint generation; without
that, a full refresh would silently skip batches 0..N. Still to do:
- failure-injection tests (failure after the Hudi commit but before the
Spark checkpoint)
- a decision on migrating legacy default_single_writer entries
- tests for truncating an empty table and truncating twice
- truncate behavior with pending compaction or clustering
- docs
- 3.2. Streaming-table full refresh: an end-to-end --full-refresh test
showing batches 0..N replay under the new identifier with no skips and history
retained.
- 3.3. MV refresh on the replacecommit truncate: switch Track A to it and
add #20405's timeline assertions (the truncate replacecommit vs the result
write).
- 3.4. Chaining across an upstream logical reset: the downstream reader
detects a truncate replacecommit (as opposed to a clustering replacecommit) and
fails with "rebuild required" instead of emitting misleading increments.
It also documents the order in which to rebuild downstream tables.
Out of scope for this issue (later work)
- DSv2 STREAMING_WRITE on HoodieInternalV2Table
- Admission control and Trigger.AvailableNow for the Hudi stream source
- Auto CDC / SCD1/2 (needs DSv2 SupportsRowLevelOperations)
- Fan-in (several flows writing one table)
- Atomic refresh across truncate → alter → write
- Full CDC propagation, including hard deletes, in chaining
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
