merlimat opened a new pull request, #4925:
URL: https://github.com/apache/bookkeeper/pull/4925
### Problem
`AuditorElector.shutdown()` runs the close of the ledger auditor manager on
the elector's single thread and waits up
to 10 seconds for it. On a bookie that is not the auditor, that thread is
waiting in
`LedgerAuditorManager.tryToBecomeAuditor()` until the bookie becomes the
auditor, so the close never runs there: every
shutdown of a non-auditor elector waits the full 10 seconds (`Failed to
close auditor manager in 10 seconds`), then
interrupts the election (`Interrupted while performing auditor election`,
logged at ERROR) and closes the manager from
the calling thread. All the bookies but the auditor pay this whenever their
auto recovery stops, e.g. at each step of
a rolling restart.
`ZkLedgerAuditorManager.close()` doesn't release a waiting candidate either,
and `tryToBecomeAuditor()` returns
normally when its loop sees the manager closed, as if the bookie had been
elected.
### Example
```java
// AuditorElector.shutdown(), master
try {
submitShutdownTask().get(10, TimeUnit.SECONDS); // queued behind the
election task
executor.shutdown();
} catch (TimeoutException e) {
log.warn().exception(e).log("Failed to close auditor manager in 10
seconds");
executor.shutdownNow(); // interrupts
tryToBecomeAuditor()
shutdownTask.run();
}
```
```java
// ZkLedgerAuditorManager.tryToBecomeAuditor(), master
while (!isClosed) {
...
CountDownLatch latch = new CountDownLatch(1);
if (null == zkc.exists(previousVote, event -> latch.countDown())) {
continue;
}
latch.await(); // close() doesn't release it
}
// returns normally once closed: the caller starts an Auditor
```
### Change
- `AuditorElector.shutdown()` closes the auditor manager first, from the
calling thread, then shuts the executor down
and waits up to 10 seconds for the election task to end (and interrupts it
after that, as before). On the auditor's
own elector the election task has already returned, so nothing changes
there.
```java
shutdownTask.run();
executor.shutdown();
if (!executor.awaitTermination(10, TimeUnit.SECONDS)) {
log.warn("Auditor election didn't stop in 10 seconds after closing the
auditor manager, interrupting");
executor.shutdownNow();
}
```
- The election task doesn't start an `Auditor` if `tryToBecomeAuditor()`
returns after the shutdown began, and when
the shutdown ends the election it neither logs an error nor submits
another shutdown task.
- `ZkLedgerAuditorManager.close()` releases a candidate waiting for the
previous vote in line, and
`tryToBecomeAuditor()` throws `IllegalStateException("Auditor manager
closed, give up to become auditor")` on a
closed manager instead of returning. Since `close()` can now run while the
candidate creates its vote, the candidate
checks again after creating it and deletes a vote `close()` might have
missed (`myVote` is now volatile), and a vote
that `close()` deleted under it no longer fails as `My vote has
disappeared` (second commit).
Another `LedgerAuditorManager` whose `close()` doesn't release a waiting
candidate keeps the old behavior: 10 seconds,
then an interrupt.
### Testing
- New `AuditorBookieTest.testShutdownFollowerElector`: stops the elector of
a bookie that isn't the auditor.
`shutdown()` must return within 5 seconds, the elector's vote must be gone
and the auditor unchanged. Fails on master
(`TimeoutException`).
- New `AuditorBookieTest.testCloseReleasesWaitingCandidate`: closes a
`ZkLedgerAuditorManager` whose
`tryToBecomeAuditor()` waits behind the running electors. The call must
fail with `IllegalStateException` within
5 seconds, and its vote must be gone. Fails on master (`TimeoutException`:
the call keeps waiting).
- The `org.apache.bookkeeper.replication` tests (19 classes) plus
`BookKeeperAdminTest`, `BookieDecommissionTest`,
`TestHttpService` and `BookieInitializationTest` pass, except 4
`BookieInitializationTest` tests that fail locally
on master too (they can't resolve the machine's hostname).
`AuditorBookieTest` and `AutoRecoveryMainTest` passed
3 more times after the second commit. Checkstyle and SpotBugs pass.
Those 23 classes on master and with this change, one run each (other test
runs were going on at the same time; the
waits removed have fixed lengths):
| Class | master (s) | this change (s) |
|---|---|---|
| `AuditorBookieTest` | 217.2 | 21.1 |
| `AuditorLedgerCheckerTest` | 212.1 | 77.6 |
| `BookieAutoRecoveryTest` | 136.3 | 68.8 |
| `BookieDecommissionTest` | 221.0 | 151.2 |
| `AuditorPeriodicCheckTest` | 269.9 | 198.8 |
| `BookKeeperAdminTest` | 59.3 | 29.3 |
| `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.4 |
| all 23 classes | 1,777 | 1,169 |
`Failed to close auditor manager in 10 seconds` appears 68 times in master's
logs of these classes, and never with
this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]