Hi guys,
I'm doing some tests and turned off 2 bookies almost simultaneously hoping
that all the ledgers would still be able to replicate since we have
ensemble and quorum size of 3.
Almost all ledgers managed to replicate using the autorecovery daemon
except for 5. What's curious about this 5 ledgers is that they are all
empty and the only node which contains data for it claims it does not exist.
Here's the ledger metadata for one of them:
ledgerID: 772
BookieMetadataFormatVersion 2
quorumSize: 3
ensembleSize: 3
length: 0
lastEntryId: -1
state: IN_RECOVERY
segment {
ensembleMember: "10.64.103.57:3181"
ensembleMember: "10.64.103.249:3181"
ensembleMember: "10.64.102.95:3181"
firstEntryId: 0
}
digestType: CRC32
password: ""
ackQuorumSize: 2
Where all nodes except 10.64.103.249 are down.
And that node contains these logs:
ERROR - [BookieReadThread-3181-10-1:ReadEntryProcessorV3@123] - No ledger
found while reading entry:-1 from ledger: 772
I don't understand how these ledgers ended in this state, is it recoverable?
I could just delete the ledgers cause they are empty too. By the way,
bookkeeper shell should have a command for deleting ledgers.
Thanks,
Sebastian
