Joy-2000 opened a new pull request, #19791: URL: https://github.com/apache/hudi/pull/19791
### Describe the issue this Pull Request addresses Under NBCC, insert overwrite reuses the deterministic bucket file id, but the replace commit it generates records that same file id as replaced. Because the file system view hides a replaced file group by file id (ignoring the replace instant), the freshly-overwritten data becomes invisible → silent data loss. ### Summary and Changelog Reject `insert overwrite` when non-blocking concurrency control (NB-CC) is enabled, and fail early — before any instant is created. **Why this is unsafe.** NB-CC requires a MOR table with a *simple* bucket index, where each bucket has one deterministic, fixed file id. `insert overwrite` reuses that same file id but generates a **replacecommit** that records the id as replaced. Because the file system view hides a replaced file group *by file id* (ignoring the replace instant), the freshly-overwritten data becomes invisible → silent data loss. **What this PR does:** - **Engine-agnostic backstop** in `BaseHoodieWriteClient#preWrite`, the single chokepoint every overwrite entry point funnels through (Spark SQL, the DataSource writer, Flink, and Hudi Streamer). Keyed on `isNonBlockingConcurrencyControl() && WriteOperationType.isOverwrite(writeOperationType)`. This closes the Hudi Streamer path, which bypasses the writer-level guards and previously only tripped the late partitioner check. - **Fail-fast guards** kept at the Spark (`HoodieSparkSqlWriter`) and Flink entry points for a friendlier early error. On Flink they're consolidated to a single call at the top of `Pipelines#hoodieStreamWrite` — the funnel for the table sink, `HoodieFlinkStreamer --op insert_overwrite`, and Sink V2 — replacing the duplicated checks in `HoodieTableSink` and `PipelinesV2`. - The rule is centralized in one validator, `WriteConcurrencyMode.checkInsertOverwriteSupported(isNbcc, isOverwrite)`, which owns the message constant and throws a single exception type (`HoodieException`). All four throw sites route through it — `preWrite`, `HoodieSparkSqlWriter`, `OptionsResolver`, and the pre-existing `BaseSparkBucketIndexBucketInfoGetter` partitioner check — replacing the previous mix of `HoodieException` and `IllegalArgumentException`. - Row-writer `bulk_insert` overwrite and case-insensitive operation strings are handled via `WriteOperationType.isOverwrite(...)` + `Locale.ROOT` normalization in both `sparkSqlInsertIntoOperation` and `deduceOverwriteConfig` (the latter previously risked deleting the table path on an uppercase operation). **Relationship to the existing check.** The Spark RDD path already rejected this since HUDI-8866 (`BaseSparkBucketIndexBucketInfoGetter`), but only *after* the requested/inflight replacecommit and the workload-profile shuffle. This PR adds (a) failing before `initTable`/`startCommit`, and (b) covering the row-writer bulk_insert overwrite path, which had no check — so the pre-existing partitioner check is superseded, not missed. ### Underlying invariant (follow-up, not in this PR) The root invariant is broader than insert overwrite: **a fixed-id bucket file group must never be listed in a replacecommit**. `delete_partition` followed by a re-insert into the same partition hits the identical trap. This PR is intentionally a narrow, safe guard for `insert overwrite` only (`isOverwrite(...)` excludes `DELETE_PARTITION`, since a standalone `delete_partition` under NB-CC is valid and loses no data). The root-cause fix — bumping the `-0` generation in `newBucketFileIdForNBCC` on replace so re-inserts land on a fresh id — is deferred to a separate ISSUE for discussion. ### Impact Users combining `insert overwrite` with non-blocking concurrency control now get a clear error at submission time instead of silent data loss. No change to any existing valid write path. ### Risk Level low Adds a validation that rejects a previously data-losing combination; existing valid writes are unaffected. ### Documentation Update The `@EnumFieldDescription` on `NON_BLOCKING_CONCURRENCY_CONTROL` now states the restriction. No site docs change needed. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added (core unit test `TestWriteConcurrencyMode`; Spark positive + negative in `TestInsertTable3`; Flink in `ITTestHoodieDataSource` / `TestOptionsResolver`) - [ ] CI passes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
