merlimat opened a new pull request, #4925:
URL: https://github.com/apache/bookkeeper/pull/4925

   ### Problem
   
   `AuditorElector.shutdown()` runs the close of the ledger auditor manager on 
the elector's single thread and waits up
   to 10 seconds for it. On a bookie that is not the auditor, that thread is 
waiting in
   `LedgerAuditorManager.tryToBecomeAuditor()` until the bookie becomes the 
auditor, so the close never runs there: every
   shutdown of a non-auditor elector waits the full 10 seconds (`Failed to 
close auditor manager in 10 seconds`), then
   interrupts the election (`Interrupted while performing auditor election`, 
logged at ERROR) and closes the manager from
   the calling thread. All the bookies but the auditor pay this whenever their 
auto recovery stops, e.g. at each step of
   a rolling restart.
   
   `ZkLedgerAuditorManager.close()` doesn't release a waiting candidate either, 
and `tryToBecomeAuditor()` returns
   normally when its loop sees the manager closed, as if the bookie had been 
elected.
   
   ### Example
   
   ```java
   // AuditorElector.shutdown(), master
   try {
       submitShutdownTask().get(10, TimeUnit.SECONDS);   // queued behind the 
election task
       executor.shutdown();
   } catch (TimeoutException e) {
       log.warn().exception(e).log("Failed to close auditor manager in 10 
seconds");
       executor.shutdownNow();                           // interrupts 
tryToBecomeAuditor()
       shutdownTask.run();
   }
   ```
   
   ```java
   // ZkLedgerAuditorManager.tryToBecomeAuditor(), master
   while (!isClosed) {
       ...
       CountDownLatch latch = new CountDownLatch(1);
       if (null == zkc.exists(previousVote, event -> latch.countDown())) {
           continue;
       }
       latch.await();   // close() doesn't release it
   }
   // returns normally once closed: the caller starts an Auditor
   ```
   
   ### Change
   
   - `AuditorElector.shutdown()` closes the auditor manager first, from the 
calling thread, then shuts the executor down
     and waits up to 10 seconds for the election task to end (and interrupts it 
after that, as before). On the auditor's
     own elector the election task has already returned, so nothing changes 
there.
   
     ```java
     shutdownTask.run();
     executor.shutdown();
     if (!executor.awaitTermination(10, TimeUnit.SECONDS)) {
         log.warn("Auditor election didn't stop in 10 seconds after closing the 
auditor manager, interrupting");
         executor.shutdownNow();
     }
     ```
   
   - The election task doesn't start an `Auditor` if `tryToBecomeAuditor()` 
returns after the shutdown began, and when
     the shutdown ends the election it neither logs an error nor submits 
another shutdown task.
   - `ZkLedgerAuditorManager.close()` releases a candidate waiting for the 
previous vote in line, and
     `tryToBecomeAuditor()` throws `IllegalStateException("Auditor manager 
closed, give up to become auditor")` on a
     closed manager instead of returning. Since `close()` can now run while the 
candidate creates its vote, the candidate
     checks again after creating it and deletes a vote `close()` might have 
missed (`myVote` is now volatile), and a vote
     that `close()` deleted under it no longer fails as `My vote has 
disappeared` (second commit).
   
   Another `LedgerAuditorManager` whose `close()` doesn't release a waiting 
candidate keeps the old behavior: 10 seconds,
   then an interrupt.
   
   ### Testing
   
   - New `AuditorBookieTest.testShutdownFollowerElector`: stops the elector of 
a bookie that isn't the auditor.
     `shutdown()` must return within 5 seconds, the elector's vote must be gone 
and the auditor unchanged. Fails on master
     (`TimeoutException`).
   - New `AuditorBookieTest.testCloseReleasesWaitingCandidate`: closes a 
`ZkLedgerAuditorManager` whose
     `tryToBecomeAuditor()` waits behind the running electors. The call must 
fail with `IllegalStateException` within
     5 seconds, and its vote must be gone. Fails on master (`TimeoutException`: 
the call keeps waiting).
   - The `org.apache.bookkeeper.replication` tests (19 classes) plus 
`BookKeeperAdminTest`, `BookieDecommissionTest`,
     `TestHttpService` and `BookieInitializationTest` pass, except 4 
`BookieInitializationTest` tests that fail locally
     on master too (they can't resolve the machine's hostname). 
`AuditorBookieTest` and `AutoRecoveryMainTest` passed
     3 more times after the second commit. Checkstyle and SpotBugs pass.
   
   Those 23 classes on master and with this change, one run each (other test 
runs were going on at the same time; the
   waits removed have fixed lengths):
   
   | Class | master (s) | this change (s) |
   |---|---|---|
   | `AuditorBookieTest` | 217.2 | 21.1 |
   | `AuditorLedgerCheckerTest` | 212.1 | 77.6 |
   | `BookieAutoRecoveryTest` | 136.3 | 68.8 |
   | `BookieDecommissionTest` | 221.0 | 151.2 |
   | `AuditorPeriodicCheckTest` | 269.9 | 198.8 |
   | `BookKeeperAdminTest` | 59.3 | 29.3 |
   | `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.4 |
   | all 23 classes | 1,777 | 1,169 |
   
   `Failed to close auditor manager in 10 seconds` appears 68 times in master's 
logs of these classes, and never with
   this change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to