[
https://issues.apache.org/jira/browse/BOOKKEEPER-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13547730#comment-13547730
]
Sijie Guo commented on BOOKKEEPER-530:
--------------------------------------
yeah, good point, Rakesh.
Compaction should not overwhelm the journal thread which might affects normal
adding operations. One by one might be better.
> data might be lost during compaction.
> -------------------------------------
>
> Key: BOOKKEEPER-530
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-530
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.1.0
> Reporter: Sijie Guo
> Assignee: Ivan Kelly
> Fix For: 4.2.0
>
> Attachments:
> 0001-BOOKKEEPER-530-data-might-be-lost-during-compaction.patch,
> 0001-BOOKKEEPER-530-data-might-be-lost-during-compaction.patch,
> 0001-BOOKKEEPER-530-data-might-be-lost-during-compaction.patch
>
>
> {code}
> try {
> entryLogger.scanEntryLog(entryLogId, new
> CompactionScanner(entryLogMeta));
> // after moving entries to new entry log, remove this old one
> removeEntryLog(entryLogId);
> } catch (IOException e) {
> LOG.info("Premature exception when compacting " + entryLogId, e);
> } finally {
> // clear compacting flag
> compacting.set(false);
> }
> {code}
> currently compaction code has a bit problem: as the code described above, old
> entry log is removed after new entries are added to new entry log, but new
> entry log might not be flushed. if failures happened after removal but before
> flush, data would be lost.
> when I implemented compaction feature in BOOKKEEPER-160, I remembered that I
> took care of letting entry go back to normal addEntry flow to reflect journal
> and index. But seems that the addEntry doesn't go thru journal, just move
> entries between entry log files w/o any flush guarantee.
> there are two ideas for this solution:
> simple one is to let compaction going to normal addEntry flow (adding entry
> to ledger storage and putting it in journal). the other one is GC thread
> either wait for ledger storage to flush in sync thread in one flush interval
> or force a ledger storage flush before removing entry log files.
> BTW, it was hard to design a test case by simulating bookie abnormally shut
> down itself after entry log files are removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
