[
https://issues.apache.org/jira/browse/BOOKKEEPER-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540818#comment-13540818
]
Sijie Guo commented on BOOKKEEPER-530:
--------------------------------------
{code}
/**
* Scanner used to do entry log compaction
*/
class EntryLogCompactionScanner implements EntryLogger.EntryLogScanner {
@Override
public boolean accept(long ledgerId) {
// bookie has no knowledge about which ledger is deleted
// so just accept all ledgers.
return true;
}
@Override
public void process(long ledgerId, long offset, ByteBuffer buffer)
throws IOException {
addEntry(buffer);
}
}
{code}
[~fpj], in compaction scanner, we just call LedgerStorage#addEntry to move
entry from old entry log file to new entry log file. we don't add this entry
again to journal during moving entries. this is what I mean.
> data might be lost during compaction.
> -------------------------------------
>
> Key: BOOKKEEPER-530
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-530
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.1.0
> Reporter: Sijie Guo
> Fix For: 4.2.0
>
>
> {code}
> try {
> entryLogger.scanEntryLog(entryLogId, new
> CompactionScanner(entryLogMeta));
> // after moving entries to new entry log, remove this old one
> removeEntryLog(entryLogId);
> } catch (IOException e) {
> LOG.info("Premature exception when compacting " + entryLogId, e);
> } finally {
> // clear compacting flag
> compacting.set(false);
> }
> {code}
> currently compaction code has a bit problem: as the code described above, old
> entry log is removed after new entries are added to new entry log, but new
> entry log might not be flushed. if failures happened after removal but before
> flush, data would be lost.
> when I implemented compaction feature in BOOKKEEPER-160, I remembered that I
> took care of letting entry go back to normal addEntry flow to reflect journal
> and index. But seems that the addEntry doesn't go thru journal, just move
> entries between entry log files w/o any flush guarantee.
> there are two ideas for this solution:
> simple one is to let compaction going to normal addEntry flow (adding entry
> to ledger storage and putting it in journal). the other one is GC thread
> either wait for ledger storage to flush in sync thread in one flush interval
> or force a ledger storage flush before removing entry log files.
> BTW, it was hard to design a test case by simulating bookie abnormally shut
> down itself after entry log files are removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
