We have observed Journal Corruption while replaying the journal log. I could be missing something but I don't see we have clear indication of where journal file ends.
In the Journal.java:scanJournal() we have multiple breaks but mostly we assume either the journal file ends in the right place or the length is zero. scanJournal() { ... while(true) { // entry start offset long offset = recLog.fc.position(); // start reading entry lenBuff.clear(); fullRead(recLog, lenBuff); if (lenBuff.remaining() != 0) { break; } lenBuff.flip(); int len = lenBuff.getInt(); if (len == 0) { break; } } ==== So there is an implicit assumption here that the journal file ends or the length is zero. But... JournalChannel.java: void preAllocIfNeeded(long size) throws IOException { if (bc.position() + size > nextPrealloc) { nextPrealloc += preAllocSize; zeros.clear(); fc.write(zeros, nextPrealloc - journalAlignSize); } } We extend the file withe prealloc size at the journalAlignSize. Based on our tunables (in our case preAllocSize:4MB journalAlignSize:512) We could extend the file and the fc.write and we are writing only the last journalAlignSize of zeros. i.e 4MB-512 can be garbage in the file. When the journal is replayed there is a potential for garbage and journal corruption indication. Being said that we haven't see this so far, so the bug may not be this obvious.. anyway just trying to see what I am missing. Here is the stack: 20191010181408.832000 [INFO ] [main] [JournalChannel.java:154] - Opening journal /sfs/sfsdata/journal/current/16cdf99de69.txn Exception in thread "main" java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:275) at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:827) at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:1047) at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:1009) at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1070) at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:141) at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:58) at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:78) at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$2(LifecycleComponentStack.java:113) at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:408) at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:113) at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:80) at org.apache.bookkeeper.server.Main.doMain(Main.java:228) at org.apache.bookkeeper.server.Main.main(Main.java:202) at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:296) -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi