oneby-wang opened a new issue, #4659: URL: https://github.com/apache/bookkeeper/issues/4659
Hi, I recently read BookKeeper journal source code, and I found that BookKeeper journal didn't use any digest to guarante journal data's integrity and consistency. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L811-L885 Due to the lack of digest validation, we don't know whether the journal log data is corrupted or not. So we can only just do replay scan until while loop breaks out or an IOException happens. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java#L823-L881 If journal data is corrupted(disk data corruption or modified by mistake), and scanning incorrect journal log data happens to produce some incorrect entries(entris that already exists in this bookie), the old entries will be replaced both in RocksDB and entryLog file, which may cause data loss. https://github.com/apache/bookkeeper/blob/2789316c18e12cbb6d17fa4a023410dbad6593a0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieImpl.java#L598C2-L607C22 I also read the ZooKeeper wal source code, and found the txnLog will append a digest after a record. So I think BookKeeper also need to do that. https://github.com/apache/zookeeper/blob/9e8296f678a9de8d9c30e3bc086a79c0cd39ef65/zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileTxnLog.java#L276-L327 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
