merlimat opened a new pull request, #4926:
URL: https://github.com/apache/bookkeeper/pull/4926
### Problem
`Auditor.shutdown()` shuts its executor down and waits up to 30 seconds for
it to terminate before interrupting the
running task (`Executor not shutting down, interrupting`). It waits, up to
the whole 30 seconds, when:
- an audit waits for the ledger replication to be enabled: the bookie check,
a lost bookie audit and a
`lostBookieRecoveryDelay` change all start with
`waitIfLedgerReplicationDisabled()`, which waits until an operator
enables it again (e.g. during a rolling restart done with the auto
recovery disabled);
- the audit of a lost bookie is delayed by `lostBookieRecoveryDelay`: a
`ScheduledThreadPoolExecutor` still runs
delayed tasks after `shutdown()`, so the auditor waits for the delay, and
when it is shorter than 30 seconds, runs
the audit after it was shut down.
### Example
```java
// Auditor.shutdown(), master
executor.shutdown(); // keeps running the delayed audit, doesn't interrupt
a blocked one
try {
while (!executor.awaitTermination(30, TimeUnit.SECONDS)) {
log.warn("Executor not shutting down, interrupting");
executor.shutdownNow();
}
```
To reproduce: disable the auto recovery (`bin/bookkeeper shell autorecovery
-disable`), stop a bookie that is not the
auditor (the auditor's audit of the lost bookie now waits for the
replication to be enabled), then stop the auditor's
bookie: it takes 30 seconds.
### Change
`shutdown()` calls `shutdownNow()` instead of `shutdown()`: it interrupts
the running task and drops the pending and
delayed ones right away, then waits for the executor to terminate as before.
```java
// Interrupt the running audit rather than wait for it: an audit can wait
for the ledger
// replication to be enabled for as long as it stays disabled, and
shutdown() would still
// run the audit delayed for a lost bookie. The next auditor runs the audits
again.
executor.shutdownNow();
```
An audit in progress is cut short instead of getting up to 30 seconds to
finish: the next auditor runs its bookie
check as soon as it starts and the periodic checks on their schedule, and
audits longer than 30 seconds were already
interrupted. Before BOOKKEEPER-496 the auditor ran on its own thread and
`shutdown()` interrupted it right away.
### Testing
- New `AuditorPeriodicCheckTest.testShutdownWithDelayedBookieAudit`: an
auditor whose audit of a lost bookie is
delayed by `lostBookieRecoveryDelay` (`Integer.MAX_VALUE`) must shut down
within 10 seconds. Fails on master
(`TimeoutException`).
- New `AuditorPeriodicCheckTest.testShutdownWithAuditWaitingForReplication`:
with the ledger replication disabled, an
auditor whose bookie check waits for it must shut down within 10 seconds.
Fails on master (`TimeoutException`).
- The `org.apache.bookkeeper.replication` tests (19 classes) plus
`BookKeeperAdminTest`, `BookieDecommissionTest`,
`TestHttpService` and `BookieInitializationTest` pass, except 4
`BookieInitializationTest` tests that fail locally
on master too (they can't resolve the machine's hostname). Checkstyle and
SpotBugs pass.
Those 23 classes on master and with this change, one run each (other test
runs were going on at the same time; the
waits removed have fixed lengths):
| Class | master (s) | this change (s) |
|---|---|---|
| `AuditorRollingRestartTest` | 96.1 | 5.9 |
| `AuditorPeriodicCheckTest` | 269.9 | 102.9 |
| `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.7 |
| all 23 classes | 1,777 | 1,522 |
`Executor not shutting down, interrupting` appears 7 times in master's logs
of these classes, and never with this
change. On its own, this change makes `AuditorLedgerCheckerTest` slower (212
to 274 s): the auditor now stops at once,
so the tests more often stop the next elector before it has become the
auditor, and that stop waits 10 seconds, which
#4925 fixes. With both changes the logs have neither wait, and the 23
classes take 919 s instead of 1,777 s:
| Class | master (s) | #4925 (s) | this change (s) | both (s) |
|---|---|---|---|---|
| `AuditorBookieTest` | 217.2 | 21.1 | 177.2 | 20.8 |
| `AuditorPeriodicCheckTest` | 269.9 | 198.8 | 102.9 | 51.1 |
| `AuditorLedgerCheckerTest` | 212.1 | 77.6 | 274.0 | 72.5 |
| `AuditorRollingRestartTest` | 96.1 | 96.1 | 5.9 | 5.9 |
| `BookieAutoRecoveryTest` | 136.3 | 68.8 | 128.6 | 68.9 |
| `BookieDecommissionTest` | 221.0 | 151.2 | 221.4 | 150.8 |
| `BookKeeperAdminTest` | 59.3 | 29.3 | 60.5 | 28.9 |
| `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.4 | 13.7 | 13.5 |
| all 23 classes | 1,777 | 1,169 | 1,522 | 919 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]