danny0405 commented on issue #20401: URL: https://github.com/apache/hudi/issues/20401#issuecomment-6463105916
Progress update as of **October 28**, using the three-phase split above. **Phase 1 — single streaming-table MVP: foundations merged; remaining patches are up for review.** - **Merged:** #20412, #20413 and #20415 cover the sink timeline-server/write-client lifetime, marker handling and schema changes. #20426 covers the MVP catalog creation and idempotent property-reconciliation work. - **Open:** #20452 addresses #20370 (fail the affected query instead of exiting the JVM), with failure-isolation and checkpoint-recovery regressions passing locally on Spark 3.5 and 4.1. - **Open:** #20455 addresses #20403 (flow-scoped settings), with compatibility documentation in #20454. The real Spark 4.1.1 CLI/Connect scenario passed locally for initial insert/upsert flows, a second input version, and an unchanged restart with exact output checks. - **Open:** #20453 addresses #20402 (Connect harness and CI). The local smoke/lifecycle tests passed, and the new [test-spark-connect-pipelines CI job](https://github.com/apache/hudi/actions/runs/40636685823/job/122645167967) has now passed too. - **To close Phase 1:** land these patches and documentation, run the combined MVP acceptance scenario on the integrated branch, and complete the remaining #20406/#20408 MVP assertions/examples. In particular, unchanged output after restart is already checked, but the CLI scenario still needs an explicit **no new Hudi commit on a no-new-data restart** assertion and the full planned upsert-dedup coverage. The broader Azure CI status is still pending on the open implementation PRs. **Phase 2 — materialized views, chaining and keyed deletes: prerequisites are progressing; the end-to-end tracks remain open.** - **Materialized views:** #20423 is still open. Its latest revision addresses the previously reported dropped write options and partition mismatch gaps, with append regression coverage on Spark 3.5/4.1. Remaining work is the MV-specific catalog handling in #20371 and repeated-refresh acceptance in #20405, using today's purge-style truncate. - **Chaining:** #20407 remains open: incremental upstream → downstream reads, starting offsets/restart behavior, clear rebuild-required errors for expired history or physical recreation, and documented source limitations. - **Keyed flows and hardening:** the remaining #20408/#20406 work covers delete markers, ordering/reinsertion and invalid configuration, resource release, fan-in rejection, and failure/restart between the Hudi commit and Spark checkpoint. The metadata-writer timeline-server lifetime follow-up also remains to be tracked and completed. **Rough wrap-up estimate:** based on the remaining scope, a tentative planning window is **November 4–6 for Phase 1**, then **another 2–3 weeks for Phase 2 (roughly November 20–27)**. This assumes prompt reviews, continued contributor availability, and no substantial issues in integrated acceptance; these are estimates, not confirmed delivery commitments. Chaining recovery and failure-injection coverage carry the most uncertainty for Phase 2. Timeline-preserving truncate and checkpoint-generation/full-refresh changes remain in **Phase 3** and are excluded from these estimates. RFC #19997 is still under review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
