Joy-2000 opened a new pull request, #19791:
URL: https://github.com/apache/hudi/pull/19791

   ### Describe the issue this Pull Request addresses
   
   Under NBCC, insert overwrite reuses the deterministic bucket file id, but 
the replace commit it generates records that same file id as replaced. Because 
the file system view hides a replaced file group by file id (ignoring the 
replace instant), the freshly-overwritten data becomes invisible → silent data 
loss.
   
   ### Summary and Changelog
   
   Reject `insert overwrite` when non-blocking concurrency control (NB-CC) is 
enabled, and fail early — before any instant is created.
   
   **Why this is unsafe.** NB-CC requires a MOR table with a *simple* bucket 
index, where each bucket has one deterministic, fixed file id. `insert 
overwrite` reuses that same file id but generates a **replacecommit** that 
records the id as replaced. Because the file system view hides a replaced file 
group *by file id* (ignoring the replace instant), the freshly-overwritten data 
becomes invisible → silent data loss.
   
   **What this PR does:**
   
   - **Engine-agnostic backstop** in `BaseHoodieWriteClient#preWrite`, the 
single chokepoint every overwrite entry point funnels through (Spark SQL, the 
DataSource writer, Flink, and Hudi Streamer). Keyed on 
`isNonBlockingConcurrencyControl() && 
WriteOperationType.isOverwrite(writeOperationType)`. This closes the Hudi 
Streamer path, which bypasses the writer-level guards and previously only 
tripped the late partitioner check.
   - **Fail-fast guards** kept at the Spark (`HoodieSparkSqlWriter`) and Flink 
entry points for a friendlier early error. On Flink they're consolidated to a 
single call at the top of `Pipelines#hoodieStreamWrite` — the funnel for the 
table sink, `HoodieFlinkStreamer --op insert_overwrite`, and Sink V2 — 
replacing the duplicated checks in `HoodieTableSink` and `PipelinesV2`.
   - The rule is centralized in one validator, 
`WriteConcurrencyMode.checkInsertOverwriteSupported(isNbcc, isOverwrite)`, 
which owns the message constant and throws a single exception type 
(`HoodieException`). All four throw sites route through it — `preWrite`, 
`HoodieSparkSqlWriter`, `OptionsResolver`, and the pre-existing 
`BaseSparkBucketIndexBucketInfoGetter` partitioner check — replacing the 
previous mix of `HoodieException` and `IllegalArgumentException`.
   - Row-writer `bulk_insert` overwrite and case-insensitive operation strings 
are handled via `WriteOperationType.isOverwrite(...)` + `Locale.ROOT` 
normalization in both `sparkSqlInsertIntoOperation` and `deduceOverwriteConfig` 
(the latter previously risked deleting the table path on an uppercase 
operation).
   
   **Relationship to the existing check.** The Spark RDD path already rejected 
this since HUDI-8866 (`BaseSparkBucketIndexBucketInfoGetter`), but only *after* 
the requested/inflight replacecommit and the workload-profile shuffle. This PR 
adds (a) failing before `initTable`/`startCommit`, and (b) covering the 
row-writer bulk_insert overwrite path, which had no check — so the pre-existing 
partitioner check is superseded, not missed.
   
   ### Underlying invariant (follow-up, not in this PR)
   
   The root invariant is broader than insert overwrite: **a fixed-id bucket 
file group must never be listed in a replacecommit**. `delete_partition` 
followed by a re-insert into the same partition hits the identical trap. This 
PR is intentionally a narrow, safe guard for `insert overwrite` only 
(`isOverwrite(...)` excludes `DELETE_PARTITION`, since a standalone 
`delete_partition` under NB-CC is valid and loses no data). The root-cause fix 
— bumping the `-0` generation in `newBucketFileIdForNBCC` on replace so 
re-inserts land on a fresh id — is deferred to a separate ISSUE for discussion.
   
   ### Impact
   
   Users combining `insert overwrite` with non-blocking concurrency control now 
get a clear error at submission time instead of silent data loss. No change to 
any existing valid write path.
   
   ### Risk Level
   
   low
   
   Adds a validation that rejects a previously data-losing combination; 
existing valid writes are unaffected.
   
   ### Documentation Update
   
   The `@EnumFieldDescription` on `NON_BLOCKING_CONCURRENCY_CONTROL` now states 
the restriction. No site docs change needed.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added (core unit test `TestWriteConcurrencyMode`; 
Spark positive + negative in `TestInsertTable3`; Flink in 
`ITTestHoodieDataSource` / `TestOptionsResolver`)
   - [ ] CI passes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to