linliu-code opened a new issue, #20213:
URL: https://github.com/apache/hudi/issues/20213
### Bug Description
**What happened:**
When HoodieStreamer (or Spark structured streaming) runs async compaction or
clustering in the same JVM, the ingest thread and the async compactor/clusterer
share one write client. That means one table service client, one
`TransactionManager`, one `LockManager` and one lock provider instance.
With `StorageBasedLockProvider` the two threads are not mutually exclusive:
- `tryLock()` returns true whenever *this provider instance* already holds
the lock (`if (actuallyHoldsLock()) return true; // Supports reentrant locks`),
whichever thread asks.
- So while the compactor thread is committing a compaction under the lock,
the ingest thread can enter a state change on the same `TransactionManager`
without waiting, for example to schedule compaction, roll back failed writes or
schedule cleaning.
- Whichever thread ends its state change first expires the lock file while
the other is still inside its critical section.
- Another writer can then acquire the lock and write the timeline
concurrently.
- Nothing logs, throws or increments a metric when this happens.
`InProcessLockProvider` and the ZooKeeper provider own the lock per thread,
so with them the second thread waits.
`TestTransactionManager#testMultiWriterTransactions` already expects the second
thread to wait, but only runs with `InProcessLockProvider`.
**What you expected:**
A second thread sharing a `TransactionManager` waits until the first ends
its state change, and it cannot release or close the first thread's lock. That
is the same behaviour as with `InProcessLockProvider`.
**Steps to reproduce:**
1. Check out #20207, which adds tests only.
2. Run
`TestTransactionManagerSharedAcrossThreads#secondThreadWaitsAndCannotReleaseHeldLock`
in `hudi-client-common`. One thread holds the lock inside a state change, and
a second thread on the same `TransactionManager` begins one.
3. With `InProcessLockProvider` the test passes. With
`StorageBasedLockProvider` both assertions fail:
- "a second thread entered the state change while another thread held the
lock";
- "another writer acquired the lock while the first thread was still
inside its state change".
`TestHoodieDeltaStreamer#testAsyncCompactorSharesIngestWriteClientAndTransactionManager`
in the same PR shows that, in continuous MOR mode with async compaction, the
async compactor holds the very same write client and `TransactionManager` as
the ingest thread. It passes in CI.
### Environment
**Hudi version:** master (`2fc6486`), and any release with
`StorageBasedLockProvider` (since HUDI-9158)
**Query engine:** Spark (HoodieStreamer continuous mode, Spark structured
streaming)
**Relevant configs:**
`hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider`,
MOR table with async compaction (or async clustering) running in the same JVM
### Logs and Stack Trace
From the CI run on #20207 (`test-common-and-other-modules`):
```
TestTransactionManagerSharedAcrossThreads.secondThreadWaitsAndCannotReleaseHeldLock:108
Multiple Failures (2 failures)
AssertionFailedError: a second thread entered the state change while
another thread held the lock ==> expected: <false> but was: <true>
AssertionFailedError: another writer acquired the lock while the first
thread was still inside its state change ==> expected: <false> but was: <true>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
