Krishen Bhan created HUDI-7503: ---------------------------------- Summary: concurrent executions of compaction plan should not corrupt dataset Key: HUDI-7503 URL: https://issues.apache.org/jira/browse/HUDI-7503 Project: Apache Hudi Issue Type: Improvement Reporter: Krishen Bhan
Currently it is not safe for 2+ writers to concurrently call `org.apache.hudi.client.BaseHoodieTableServiceClient#compact` on the same compaction instant. This is since one writer might execute the instant and create an inflight, while the other writer sees the inflight and tries to roll it back before re-attempting to execute it (since it will assume said inflight was a previously failed compaction attempt). This logic should be updated such that only one writer will actually execute the compaction plan at a time (and the others will fail/abort). One approach is to use a transaction (base table lock) in conjunction with heartbeating, to ensure that the writer triggers a heartbeat before executing compaction, and any concurrent writers will use the heartbeat to check wether the compaction is currently being executed by another writer. Specifically , the compact API should execute the following steps # Get the instant to compact C (as usual) # Start a transaction # Checks if C has an active heartbeat, if so finish transaction and throw exception # Start a heartbeat for C (this will implicitly re-start the heartbeat if it has been started before by another job) # Finish transaction # Run the existing compact API logic on C # If execution succeeds, clean up heartbeat file . If it fails do nothing (as the heartbeat will anyway be automatically expired later). Note that this approach only holds the table lock temporarily, when checking/starting the heartbeat -- This message was sent by Atlassian Jira (v8.20.10#820010)