Joy-2000 opened a new issue, #19715: URL: https://github.com/apache/hudi/issues/19715
### Context Split out from the broader lock-scope discussion in #19587, to keep the design and testing focused. ### Problem `commitStats`, `completeClustering`, `completeCompaction`, and `completeLogCompaction` perform several operations while completing a write or table service, including `resolveWriteConflict`, `finalizeWrite`, `writeTableMetadata`, and `saveAsComplete`. `finalizeWrite` removes speculative or invalid files based on the markers for the current instant. When a write touches many partitions or files, `DirectWriteMarkers` may issue a large number of filesystem `list` operations, so `finalizeWrite` can hold the transaction lock for a long time when the filesystem is busy. ### Proposal Move `finalizeWrite` outside the transaction boundary, keeping conflict resolution, Metadata Table updates, and timeline completion inside the lock. The sequence likes: > finalize first → acquire the transaction lock → reload current table/timeline state → perform conflict resolution, Metadata Table update, and timeline completion in one critical section. ### Potential risk It must **not** be split as `conflict-resolution-under-lock → unlock → finalize → relock`, because a commit completing in that gap invalidates the conflict check. `finalizeWrite` is **not** read-only: marker reconciliation deletes invalid data files. Moving it outside the lock could race with rollback, cleaning, or another writer operating on the same files. We do not currently see an obvious correctness dependency requiring `finalizeWrite` itself to run under the lock, but this must be verified for all marker implementations and failure-recovery paths. ### Expected benefit Reduce lock duration caused by filesystem operations during `finalizeWrite`, especially for large writes or when HDFS is under heavy load. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
