danny0405 opened a new pull request, #20754: URL: https://github.com/apache/hudi/pull/20754
### Describe the issue this Pull Request addresses Closes #20405. Closes #20752. Closes #20753. Part of #20401. Adding a column to an SDP materialized view succeeds, but the next unchanged refresh purges the timeline and falls back to the original table schema. The result write then fails with INSERT_COLUMN_ARITY_MISMATCH. Both managed and external target acceptance reproduced this on master. This completes the remaining MV acceptance in one follow-up to merged #20494 and #20675. Closure is for the agreed purge-style scope; timeline-preserving truncate and checkpoint-generation isolation remain deferred together under #20404 / #20378. ### Summary and Changelog - Resolve the current schema before full-table purge and use it when initializing the empty table. Clear metadata-partition state through the new table config so the old CREATE_SCHEMA is not written back. - Add COW/MOR × partitioned/unpartitioned SQL regressions: add a column, truncate twice including an empty table, verify the persisted schema and purged timeline, then insert and read the evolved row. - Extend the existing materialized_views.py suite with SQL MV schema evolution, unchanged refresh, partition/key rejection, and failures before purge, during reconciliation after purge, and during result writing. Assert exact rows, schema, identity, location/type and completed timeline state, then retry. - Add an opt-in --external manual run of the same matrix against a pre-created explicitly located target. Retain generated SQL, expected rows, timeline/schema evidence and CLI/server diagnostics. No CI jobs are added. - Document the destructive reconciliation failure window and retry procedure. No external code was copied. ### Impact Repairs schema loss for ordinary full-table TRUNCATE and SDP MV refresh. Purge still removes data/history/savepoints and resets metadata state; it is non-transactional. Partition truncate is unchanged. No new public API, configuration or storage format. The CLI acceptance uses COW with metadata indexing disabled. It does not establish MOR/metadata-enabled SDP acceptance, atomic refresh, or streaming-table full refresh. ### Risk Level medium. Shared SQL truncate now resolves schema before deletion and retains that schema in the reinitialized table config. Validation on JDK 17 / Spark 4.1 / Scala 2.13: - Reactor package, Checkstyle, Scalastyle and RAT checks passed. - TestTruncateTable: 10 passed, 0 failed, including all four new schema regressions. - Real Spark Connect 4.1.1 / spark-pipelines CLI runs passed with python -O for managed targets and the manual --external variant. Each passed the original four keyed/keyless refreshes and the full schema/failure/retry matrix, including empty recomputation and subsequent nonempty refresh. - Python compilation and git diff --check passed. Spark 3.5 and remote CI have not been run for this revision. Failures inside a partially completed filesystem purge are not injected; no rollback after deletion begins is promised. ### Documentation Update Updated hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md with schema retention, the observed failure boundaries, exact retry expectations and the manual external-target command. This repairs the existing truncate contract without introducing a new feature or configuration. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
