vamshipasunuru1 opened a new issue, #20562:
URL: https://github.com/apache/hudi/issues/20562

   ### Bug Description
   
   **What happened:**
   In multi-writer mode (OCC with the LAZY failed-writes policy), two async 
table-service jobs ran the same compaction instant `T` at the same time:
   
   1. Job A started executing compaction `T`. Because of an infra issue, A's 
heartbeat for `T` expired, but A's executors kept running.
   2. Job B saw the expired heartbeat and rolled `T` back, deleting the base 
files A had written so far.
   3. Job A finished and committed `T` through `completeCompaction`, with write 
stats pointing to the deleted files.
   
   On the metadata table, this left the record index pointing at file slices 
that don't exist. Once the cleaner removed the previous slice, record-index 
entries were lost, and **upserts re-inserted existing keys as duplicates.**
   
   Why nothing blocked step 3:
   - `completeCompaction` and `completeLogCompaction` don't check the 
heartbeat, unlike the write commit path (`BaseHoodieWriteClient#commitStats` → 
`WriterHeartbeatUtils.abortIfHeartbeatExpired`).
   - OCC conflict resolution skips compaction commits, and a rollback of the 
same instant isn't treated as a conflict.
   - The lock covers only the commit critical section, not the time the 
compaction is running.
   
   **What you expected:**
   A writer whose heartbeat has expired may already have been rolled back, so 
it should fail at commit instead of committing.
   
   **Steps to reproduce:**
   1. Configure OCC, `hoodie.write.failed.writes.cleaner.policy=LAZY` and a 
lock provider on a table with the metadata table enabled, then schedule 
compaction `T`.
   2. Writer A: start executing `T`, block it before `completeCompaction`, and 
let its heartbeat expire.
   3. Writer B: trigger the failed-writes rollback, which rolls back `T`.
   4. Unblock A. `completeCompaction` succeeds and references base files that 
no longer exist.
   
   
   **Proposed fix:**
   In `completeCompaction` and `completeLogCompaction`, call 
`WriterHeartbeatUtils.abortIfHeartbeatExpired(...)` after `beginStateChange` 
when a heartbeat was started for the instant.
   
   ### Environment
   
   **Environment:**
   - Hudi 1.2; the same code path is on master 
(`BaseHoodieTableServiceClient#completeCompaction`, `#completeLogCompaction`)
   - Spark 3.x, OCC, ZooKeeper lock provider
   
   
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to