Radiancebobo opened a new issue, #4872:
URL: https://github.com/apache/bookkeeper/issues/4872
## 1. Feature requested
BookKeeper AutoRecovery should recognize failed-Bookie underreplication
tasks for
ledgers whose `writeQuorumSize == 1` and avoid entering an endless recovery
loop
when the failed Bookie is permanently unavailable.
For these ledgers, each entry has only one data source. Once that source is
permanently lost, AutoRecovery cannot reconstruct the missing data from
another
Bookie. The requested behavior is therefore to:
- avoid publishing new failed-Bookie underreplication tasks for known
single-replica ledgers;
- remove matching historical failed-Bookie tasks created before this behavior
was introduced;
- always release the underreplication lock during the cleanup path;
- preserve placement-policy repair tasks, including `writeQuorumSize == 1`
tasks with an empty replica list; and
- expose dedicated Auditor and ReplicationWorker metrics so operators can
identify possible data loss and take manual recovery action.
This feature must not claim to recover data that has no surviving replica. It
should only stop the impossible automatic recovery path and make the
condition
visible to operators.
## 2. Importance and workarounds
Importance: **must-have** for deployments that create ledgers with
`writeQuorumSize == 1`; **should-have** otherwise.
Without this behavior, a permanently lost Bookie leaves an underreplication
task
that the Worker repeatedly locks, attempts to open or read, fails to recover,
releases the lock, and retries. This consumes Worker, BookKeeper client, and
ZooKeeper resources, produces noisy failure logs, and can delay processing of
ledgers that are actually recoverable.
There is no safe automatic workaround for the lost data itself. Operators can
restore the original Bookie or recover data from backups, but those options
may
not be available. Disabling AutoRecovery or manually deleting the
underreplication task can stop the retry noise, but disabling AutoRecovery
also
stops unrelated recovery and deleting the task can hide a data-loss
condition.
## 3. Proposed use case
A cluster contains a ledger created with `writeQuorumSize == 1`. The ledger's
only Bookie fails permanently. The Auditor detects the failed Bookie, reads
the
ledger metadata, and identifies that automatic reconstruction is impossible.
Instead of creating a task that will fail forever, the Auditor skips
publication
and increments a dedicated metric.
During an upgrade, the cluster may already contain persistent
underreplication
znodes created by an older BookKeeper version. When a Worker consumes one of
these historical tasks, it should inspect the task shape and ledger metadata,
clean up only failed-Bookie tasks for single-replica ledgers, release the
lock,
and report a skipped outcome. A placement-policy task with an empty replica
list
must continue through the normal replication path.
The metrics should allow operators to alert on skipped ledgers and decide
whether to restore the original Bookie, use backups, or apply an
application-specific recovery procedure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
