Davis-Zhang-Onehouse opened a new pull request, #19720: URL: https://github.com/apache/hudi/pull/19720
### Describe the issue this Pull Request addresses Under OCC, `SimpleConcurrentFileWritesConflictResolutionStrategy.hasConflict` only reports a conflict when two operations share a (partition, file id) pair. Two `INSERT_OVERWRITE` writers that both planned against an empty partition replace nothing and write fresh file ids, so both commit and every row of the partition is present twice; clean, clustering and compaction then keep both copies. `INSERT_OVERWRITE` has partition semantics but OCC only records it as file ids. ### Summary and Changelog Two overwrite operations (`INSERT_OVERWRITE` / `INSERT_OVERWRITE_TABLE`) that touch a common partition now conflict once the other one has completed, regardless of file ids. Scope, deliberately narrow: - only when the other overwrite is already completed (first to commit wins, as elsewhere in OCC). Matching pending overwrites too would make both writers fail each other and would let an abandoned pending overwrite block every later overwrite of that partition until lazy cleaning rolls it back; - only overwrite against overwrite. A plain insert that lands in a partition a concurrent overwrite completed on keeps today's file-level semantics. Known conservative case: an `INSERT_OVERWRITE` of partition P that planned after a concurrent `INSERT_OVERWRITE_TABLE` completed is now rejected as well (the reverse ordering already was, via file ids). Not in this PR: `BucketIndexConcurrentFileWritesConflictResolutionStrategy` overrides `hasConflict` entirely and does not get this rule. Changes: - `SimpleConcurrentFileWritesConflictResolutionStrategy.hasConflict`: partition-level check for a completed overwrite of a common partition. - `TestSimpleConcurrentFileWritesConflictResolutionStrategy`: predicate-level test (same partition / pending other / different partition / non-overwrite). - `TestConcurrentInsertOverwriteSamePartition`: two `SparkRDDWriteClient`s overwriting the same empty partition, plus a populated-partition contrast. - `TestConcurrentInsertOverwritePartition`: two Spark SQL sessions running `INSERT OVERWRITE ... PARTITION` concurrently. No code copied. ### Impact A second concurrent `INSERT OVERWRITE` of the same partition now fails with `HoodieWriteConflictException` instead of silently duplicating the partition's data. No API or config change. Inherited by strategies that subclass `SimpleConcurrentFileWritesConflictResolutionStrategy` without overriding `hasConflict`. ### Risk Level Low. Tests at three levels, all red on the old predicate (see Changelog). `TestHoodieClientMultiWriter` passes unchanged. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
