linliu-code opened a new pull request, #20207:
URL: https://github.com/apache/hudi/pull/20207
### Describe the issue this Pull Request addresses
Reproduces #20213 (tests only; the fix will be a separate PR).
This PR only adds tests. They show that two threads sharing one
`TransactionManager` can both be inside a state change at once when the lock
provider is `StorageBasedLockProvider`, and that the second thread then
releases the first thread's lock. I'm opening it as a draft so we can agree the
problem is real before discussing a fix. The tests fail on master on purpose.
How two threads end up sharing one `TransactionManager`:
- HoodieStreamer gives its ingest write client to the in-process async
compactor and clusterer (`HoodieStreamer#onInitializingWriteClient` creates
`SparkAsyncCompactService(hoodieSparkContext, writeClient)`, and
`HoodieSparkCompactor` calls `writeClient.compact` and
`writeClient.commitCompaction` on that object). Spark streaming does the same
in `HoodieStreamingSink`.
- The compactor thread commits through the write client's table service
client: `completeCompaction`, then `txnManager.beginStateChange`.
- The ingest thread also uses that table service client to schedule
compaction and clustering, to roll back failed writes and to schedule cleaning
(`scheduleTableServiceInternal`, `rollbackFailedWrites`, `scheduleCleaning`).
So both threads take the lock through the same `TransactionManager`,
`LockManager` and lock provider instance.
The write client has a second `TransactionManager` (the one in
`BaseHoodieClient`), which ingest commits use, so an ingest commit still
excludes a compaction commit correctly. The overlap is between the async
compactor or clusterer and the ingest thread's table-service calls: scheduling,
rollback of failed writes and clean scheduling.
Why only `StorageBasedLockProvider`:
- `StorageBasedLockProvider.tryLock()` returns true whenever *this provider
instance* already holds the lock (`if (actuallyHoldsLock()) return true; //
Supports reentrant locks`), whichever thread asks. The second thread therefore
enters without waiting.
- The lock can then be released while the other thread is still committing.
Whether it is depends on the owner instants the two threads pass and on which
one ends first, because `TransactionManager.reset` lets a caller with an empty
instant end the state change. The test reproduces the case below: the second
thread passes an empty instant and ends first.
- `InProcessLockProvider` and the ZooKeeper provider own the lock per
thread, so the second thread waits.
`TestTransactionManager#testMultiWriterTransactions` already expects that wait,
but it only runs with `InProcessLockProvider`.
#### Illustration
How the two threads come to share one lock provider instance
(HoodieStreamer, continuous mode, async compaction):
```
HoodieStreamer JVM
ingest thread (StreamSync) async compactor thread
(AsyncCompactService)
│ writeClient.scheduleCompaction() │ writeClient.compact()
/ commitCompaction()
└──────────────┐ ┌───────────┘
▼ ▼
writeClient (one object, handed to the compactor in
onInitializingWriteClient)
└─► tableServiceClient ─► TransactionManager ─►
LockManager ─► StorageBasedLockProvider
(one instance, one owner id)
│
▼
lock file in storage
```
What happens when both threads take the lock (what
TestTransactionManagerSharedAcrossThreads reproduces):
```
time compactor thread ingest thread
lock file
t1 beginStateChange(compaction)
written: held by owner X
t2 finalizeWrite, write metadata table ... beginStateChange(schedule /
rollback)
provider.tryLock(): "this
instance
already holds the lock" →
returns true
t3 (still committing) ** both threads inside the
critical section **
t4 (still committing) endStateChange() →
provider.unlock() ──► expired: released
t5 (still committing, no lock held)
another writer W2 ► written: held by W2
t6 completeInflightCompaction() ◄── writes the timeline while W2 holds
the lock
t7 endStateChange() → nothing left to release, no error, no metric
```
With InProcessLockProvider (or ZooKeeper) the ingest thread waits at t2
until the compactor's endStateChange, because those providers own the lock per
thread.
Consequence: mutual exclusion is broken. Work that assumes it is the only
writer (completing the compaction and writing the metadata table, scheduling
table services, rolling back failed writes) can run concurrently with another
writer. In the reproduced case nothing logs, throws or increments a metric. In
one other ordering (the first thread passes an empty instant, the second passes
one, and the first ends first) the first thread's `endStateChange` throws
`NoSuchElementException` from `TransactionManager.reset`, so that case is at
least visible.
### Summary and Changelog
Tests only, no production code change:
-
`TestTransactionManagerSharedAcrossThreads#secondThreadWaitsAndCannotReleaseHeldLock`,
parameterized over the lock provider:
- One thread holds the lock inside a compaction state change.
- A second thread on the same `TransactionManager` begins a state change.
- The test asserts that the second thread did not enter, and that an
independent writer cannot acquire the lock while the first thread is still
inside.
- `InProcessLockProvider` is the control and passes.
`StorageBasedLockProvider` fails both assertions on master.
- `InMemoryStorageBasedLockProvider` (test helper):
`StorageBasedLockProvider` with its locking logic unchanged, backed by
in-memory storage with the same conditional-write semantics as S3, GCS and
Azure.
-
`TestHoodieDeltaStreamer#testAsyncCompactorSharesIngestWriteClientAndTransactionManager`:
in continuous MOR mode with async compaction, the async compactor holds the
very same write client and `TransactionManager` as the ingest thread.
How the tests were run: TestTransactionManagerSharedAcrossThreads ran
locally against master. The InProcessLockProvider case passes; the
StorageBasedLockProvider case fails both assertions ("a second thread entered
the state change while another thread held the lock" and "another writer
acquired the lock while the first thread was still inside its state change").
The TestHoodieDeltaStreamer case needs Docker for the class's test containers
and was not run locally; CI runs it here for the first time.
### Impact
None; tests only. On master, the `StorageBasedLockProvider` case of the new
test fails until there is a fix.
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]