linliu-code opened a new pull request, #20207:
URL: https://github.com/apache/hudi/pull/20207

   ### Describe the issue this Pull Request addresses
   
   Reproduces #20213 (tests only; the fix will be a separate PR).
   
   This PR only adds tests. They show that two threads sharing one 
`TransactionManager` can both be inside a state change at once when the lock 
provider is `StorageBasedLockProvider`, and that the second thread then 
releases the first thread's lock. I'm opening it as a draft so we can agree the 
problem is real before discussing a fix. The tests fail on master on purpose.
   
   How two threads end up sharing one `TransactionManager`:
   - HoodieStreamer gives its ingest write client to the in-process async 
compactor and clusterer (`HoodieStreamer#onInitializingWriteClient` creates 
`SparkAsyncCompactService(hoodieSparkContext, writeClient)`, and 
`HoodieSparkCompactor` calls `writeClient.compact` and 
`writeClient.commitCompaction` on that object). Spark streaming does the same 
in `HoodieStreamingSink`.
   - The compactor thread commits through the write client's table service 
client: `completeCompaction`, then `txnManager.beginStateChange`.
   - The ingest thread also uses that table service client to schedule 
compaction and clustering, to roll back failed writes and to schedule cleaning 
(`scheduleTableServiceInternal`, `rollbackFailedWrites`, `scheduleCleaning`).
   
   So both threads take the lock through the same `TransactionManager`, 
`LockManager` and lock provider instance.
   
   The write client has a second `TransactionManager` (the one in 
`BaseHoodieClient`), which ingest commits use, so an ingest commit still 
excludes a compaction commit correctly. The overlap is between the async 
compactor or clusterer and the ingest thread's table-service calls: scheduling, 
rollback of failed writes and clean scheduling.
   
   Why only `StorageBasedLockProvider`:
   - `StorageBasedLockProvider.tryLock()` returns true whenever *this provider 
instance* already holds the lock (`if (actuallyHoldsLock()) return true; // 
Supports reentrant locks`), whichever thread asks. The second thread therefore 
enters without waiting.
   - The lock can then be released while the other thread is still committing. 
Whether it is depends on the owner instants the two threads pass and on which 
one ends first, because `TransactionManager.reset` lets a caller with an empty 
instant end the state change. The test reproduces the case below: the second 
thread passes an empty instant and ends first.
   - `InProcessLockProvider` and the ZooKeeper provider own the lock per 
thread, so the second thread waits. 
`TestTransactionManager#testMultiWriterTransactions` already expects that wait, 
but it only runs with `InProcessLockProvider`.
   
   #### Illustration
   
   How the two threads come to share one lock provider instance 
(HoodieStreamer, continuous mode, async compaction):
   
   ```
   HoodieStreamer JVM
     ingest thread (StreamSync)                 async compactor thread 
(AsyncCompactService)
           │ writeClient.scheduleCompaction()          │ writeClient.compact() 
/ commitCompaction()
           └──────────────┐                ┌───────────┘
                          ▼                ▼
                   writeClient  (one object, handed to the compactor in 
onInitializingWriteClient)
                          └─► tableServiceClient ─► TransactionManager ─► 
LockManager ─► StorageBasedLockProvider
                                                                                
        (one instance, one owner id)
                                                                                
               │
                                                                                
               ▼
                                                                                
   lock file in storage
   ```
   
   What happens when both threads take the lock (what 
TestTransactionManagerSharedAcrossThreads reproduces):
   
   ```
   time  compactor thread                         ingest thread                 
             lock file
    t1   beginStateChange(compaction)                                           
             written: held by owner X
    t2   finalizeWrite, write metadata table ...  beginStateChange(schedule / 
rollback)
                                                    provider.tryLock(): "this 
instance
                                                    already holds the lock" → 
returns true
    t3   (still committing)                       ** both threads inside the 
critical section **
    t4   (still committing)                       endStateChange() → 
provider.unlock() ──►  expired: released
    t5   (still committing, no lock held)                                
another writer W2 ►  written: held by W2
    t6   completeInflightCompaction()  ◄── writes the timeline while W2 holds 
the lock
    t7   endStateChange() → nothing left to release, no error, no metric
   ```
   
   With InProcessLockProvider (or ZooKeeper) the ingest thread waits at t2 
until the compactor's endStateChange, because those providers own the lock per 
thread.
   
   Consequence: mutual exclusion is broken. Work that assumes it is the only 
writer (completing the compaction and writing the metadata table, scheduling 
table services, rolling back failed writes) can run concurrently with another 
writer. In the reproduced case nothing logs, throws or increments a metric. In 
one other ordering (the first thread passes an empty instant, the second passes 
one, and the first ends first) the first thread's `endStateChange` throws 
`NoSuchElementException` from `TransactionManager.reset`, so that case is at 
least visible.
   
   ### Summary and Changelog
   
   Tests only, no production code change:
   - 
`TestTransactionManagerSharedAcrossThreads#secondThreadWaitsAndCannotReleaseHeldLock`,
 parameterized over the lock provider:
     - One thread holds the lock inside a compaction state change.
     - A second thread on the same `TransactionManager` begins a state change.
     - The test asserts that the second thread did not enter, and that an 
independent writer cannot acquire the lock while the first thread is still 
inside.
     - `InProcessLockProvider` is the control and passes. 
`StorageBasedLockProvider` fails both assertions on master.
   - `InMemoryStorageBasedLockProvider` (test helper): 
`StorageBasedLockProvider` with its locking logic unchanged, backed by 
in-memory storage with the same conditional-write semantics as S3, GCS and 
Azure.
   - 
`TestHoodieDeltaStreamer#testAsyncCompactorSharesIngestWriteClientAndTransactionManager`:
 in continuous MOR mode with async compaction, the async compactor holds the 
very same write client and `TransactionManager` as the ingest thread.
   
   How the tests were run: TestTransactionManagerSharedAcrossThreads ran 
locally against master. The InProcessLockProvider case passes; the 
StorageBasedLockProvider case fails both assertions ("a second thread entered 
the state change while another thread held the lock" and "another writer 
acquired the lock while the first thread was still inside its state change"). 
The TestHoodieDeltaStreamer case needs Docker for the class's test containers 
and was not run locally; CI runs it here for the first time.
   
   ### Impact
   
   None; tests only. On master, the `StorageBasedLockProvider` case of the new 
test fails until there is a fix.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to